+
+ [](https://hits.seeyoufarm.com)
+
+
+
@@ -201,7 +208,46 @@ https://blogshine.tistory.com/345
**상세 내용 링크 : ([글 링크](https://blogshine.tistory.com/664))**
---
-## 5. 인증, 인가를 비즈니스 로직으로부터 분리하기
+# 5. Bulk Query를 통한 성능 개선
+
+**문제 상황**
+
+- 공지를 주기적으로 저장하고 삭제하는 과정이 하루에도 수십번 반복하는 이 앱의 핵심 로직중 하나입니다.
+- 문제는 이 과정에서 발생하는 쿼리가 너무 많다는 것입니다.
+ 쿼리로그를 찍어본 결과 save와 delete 모두 한방 쿼리가 아니라 여러번의 쿼리가 나가는 것을 확인했습니다.
+
+**문제 해결**
+
+
+
+
+다음과 같이 수정하여 해결할 수 있었습니다.
++ Insert의 경우 : JdbcTemplate.batchUpdate() 사용
++ delete의 경우 : queryDsl의 in 쿼리 사용
+
+
+
+### 5-1) Insert 해결책
+
+해결책은 2가지가 존재했습니다.
+1. Table Id strategy를 SEQUENCE로 변경하고 Batch 작업
+2. JdbcTemplate.batchUpdate() 사용
+
+MySQL과 MariaDB의 Table Id 전략은 대부분이 IDENTITY 전략을 사용하기도 하고, 저희는 이미 Id 전략을 IDENTITY 전략으로 사용하고 있었기에 Id전략 자체를 변경하기에는 무리가 있었습니다.
+또한, Jdbc를 사용하는 것이 성능상 더 뛰어나다는 결과를 확인했습니다. [출처](https://homoefficio.github.io/2020/01/25/Spring-Data에서-Batch-Insert-최적화/#)
+
+
+
+
+
+
+
+### 5-2) Delete 해결책
+이미 프로젝트에서 queryDsl를 사용하고 있어 이를 이용하는 것이 가장 간단했기 때문에 queryDsl의 delete in 쿼리를 사용하여 해결했습니다.
+
+---
+
+## 6. 인증, 인가를 비즈니스 로직으로부터 분리하기
**문제 상황**
@@ -219,7 +265,7 @@ https://blogshine.tistory.com/345
---
-## 6. 흔하디 흔한 N+1 쿼리 개선기
+## 7. 흔하디 흔한 N+1 쿼리 개선기
원래 로직에서는 사용자의 Category 이름 목록을 가져오기 위해서 다음과 같이 처리가 되고 잇었습니다!
@@ -277,7 +323,7 @@ public List
getCategoryNamesFromCategories(List categories) {
쿼리가 총 1 + 2N 만큼 발생중이다.
-### 6 - 1) 변경 전 쿼리
+### 7 - 1) 변경 전 쿼리
```bash
Hibernate:
@@ -334,7 +380,7 @@ Connection: keep-alive
N+1 문제로 User한번 조회하는데 위와 같이 쿼리가 3번 나가게 됨
-### 6 - 2) 변경 후
+### 7 - 2) 변경 후
변경 후 한방 쿼리로 조회 끝
```java
@@ -351,7 +397,7 @@ public List getUserCategoryNamesByToken(String token) {
___
-## 7. Test Container를 통한 테스트의 멱등성 보장하기
+## 8. Test Container를 통한 테스트의 멱등성 보장하기
테스트와, 실제 운영 DB를 둘다 MariaDB 환경으로 사용하여 문제가 발생할 일이 없다 생각했었습니다.
하지만, utf8과 같은 인코딩 방식이 로컬과 프로덕션이 달라 문제가 발생하였으며, 이또한 테스트 환경에서 걸러내지 못한 것이 문제라 생각하였습니다.
@@ -360,7 +406,7 @@ ___
---
-## 8. CI / 정적분석기(SonarCloud, jacoco)를 사용한 코드 컨벤션에 대한 코드리뷰 자동화
+## 9. CI / 정적분석기(SonarCloud, jacoco)를 사용한 코드 컨벤션에 대한 코드리뷰 자동화
**문제 상황**
@@ -377,7 +423,7 @@ ___
---
-## 9. 서버 모니터링
+## 10. 서버 모니터링
**문제 상황**
@@ -485,6 +531,5 @@ dev, local 환경에서는 단순히 ddl을 create-drop 또는 update 옵션을
4) [Github Actions, CodeDeploy, Nginx 로 무중단 배포하기 - 4](https://blogshine.tistory.com/430)
-
-[](https://hits.seeyoufarm.com)
+
diff --git a/build.gradle b/build.gradle
index 4241aecc..d075c2b6 100644
--- a/build.gradle
+++ b/build.gradle
@@ -9,7 +9,7 @@ plugins {
id 'io.spring.dependency-management' version '1.0.11.RELEASE'
id 'com.ewerk.gradle.plugins.querydsl' version "1.0.10"
id 'java'
- id 'org.asciidoctor.convert' version '1.5.3'
+ id 'org.asciidoctor.jvm.convert' version "3.3.2"
id 'org.flywaydb.flyway' version '9.16.1' // flyway gradle plugin 의존성
id 'org.sonarqube' version '3.5.0.2730' // sonarqube gradle plugin 의존성
id 'jacoco' // jacoco gradle plugin 의존성
@@ -17,9 +17,10 @@ plugins {
group = 'com.kustacks'
version = '1.1.2'
-sourceCompatibility = '11'
+sourceCompatibility = '17'
configurations {
+ asciidoctorExtensions
compileOnly {
extendsFrom annotationProcessor
}
@@ -89,7 +90,7 @@ dependencies {
implementation 'com.google.firebase:firebase-admin:8.1.0'
// API Docs
- asciidoctor 'org.springframework.restdocs:spring-restdocs-asciidoctor'
+ asciidoctorExtensions 'org.springframework.restdocs:spring-restdocs-asciidoctor'
testImplementation 'org.springframework.restdocs:spring-restdocs-mockmvc'
// Test
diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar
index 7454180f..e708b1c0 100644
Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
index 549d8442..e750102e 100644
--- a/gradle/wrapper/gradle-wrapper.properties
+++ b/gradle/wrapper/gradle-wrapper.properties
@@ -1,5 +1,5 @@
distributionBase=GRADLE_USER_HOME
distributionPath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-6.9-bin.zip
+distributionUrl=https\://services.gradle.org/distributions/gradle-7.3-bin.zip
zipStoreBase=GRADLE_USER_HOME
zipStorePath=wrapper/dists
diff --git a/gradlew b/gradlew
index 3da45c16..4f906e0c 100755
--- a/gradlew
+++ b/gradlew
@@ -1,7 +1,7 @@
-#!/bin/sh
+#!/usr/bin/env sh
#
-# Copyright ? 2015-2021 the original authors.
+# Copyright 2015 the original author or authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@@ -17,101 +17,67 @@
#
##############################################################################
-#
-# Gradle start up script for POSIX generated by Gradle.
-#
-# Important for running:
-#
-# (1) You need a POSIX-compliant shell to run this script. If your /bin/sh is
-# noncompliant, but you have some other compliant shell such as ksh or
-# bash, then to run this script, type that shell name before the whole
-# command line, like:
-#
-# ksh Gradle
-#
-# Busybox and similar reduced shells will NOT work, because this script
-# requires all of these POSIX shell features:
-# * functions;
-# * expansions ?$var?, ?${var}?, ?${var:-default}?, ?${var+SET}?,
-# ?${var#prefix}?, ?${var%suffix}?, and ?$( cmd )?;
-# * compound commands having a testable exit status, especially ?case?;
-# * various built-in commands including ?command?, ?set?, and ?ulimit?.
-#
-# Important for patching:
-#
-# (2) This script targets any POSIX shell, so it avoids extensions provided
-# by Bash, Ksh, etc; in particular arrays are avoided.
-#
-# The "traditional" practice of packing multiple parameters into a
-# space-separated string is a well documented source of bugs and security
-# problems, so this is (mostly) avoided, by progressively accumulating
-# options in "$@", and eventually passing that to Java.
-#
-# Where the inherited environment variables (DEFAULT_JVM_OPTS, JAVA_OPTS,
-# and GRADLE_OPTS) rely on word-splitting, this is performed explicitly;
-# see the in-line comments for details.
-#
-# There are tweaks for specific operating systems such as AIX, CygWin,
-# Darwin, MinGW, and NonStop.
-#
-# (3) This script is generated from the Groovy template
-# https://github.com/gradle/gradle/blob/master/subprojects/plugins/src/main/resources/org/gradle/api/internal/plugins/unixStartScript.txt
-# within the Gradle project.
-#
-# You can find Gradle at https://github.com/gradle/gradle/.
-#
+##
+## Gradle start up script for UN*X
+##
##############################################################################
# Attempt to set APP_HOME
-
# Resolve links: $0 may be a link
-app_path=$0
-
-# Need this for daisy-chained symlinks.
-while
- APP_HOME=${app_path%"${app_path##*/}"} # leaves a trailing /; empty if no leading path
- [ -h "$app_path" ]
-do
- ls=$( ls -ld "$app_path" )
- link=${ls#*' -> '}
- case $link in #(
- /*) app_path=$link ;; #(
- *) app_path=$APP_HOME$link ;;
- esac
+PRG="$0"
+# Need this for relative symlinks.
+while [ -h "$PRG" ] ; do
+ ls=`ls -ld "$PRG"`
+ link=`expr "$ls" : '.*-> \(.*\)$'`
+ if expr "$link" : '/.*' > /dev/null; then
+ PRG="$link"
+ else
+ PRG=`dirname "$PRG"`"/$link"
+ fi
done
-
-APP_HOME=$( cd "${APP_HOME:-./}" && pwd -P ) || exit
+SAVED="`pwd`"
+cd "`dirname \"$PRG\"`/" >/dev/null
+APP_HOME="`pwd -P`"
+cd "$SAVED" >/dev/null
APP_NAME="Gradle"
-APP_BASE_NAME=${0##*/}
+APP_BASE_NAME=`basename "$0"`
# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"'
# Use the maximum available, or set MAX_FD != -1 to use that value.
-MAX_FD=maximum
+MAX_FD="maximum"
warn () {
echo "$*"
-} >&2
+}
die () {
echo
echo "$*"
echo
exit 1
-} >&2
+}
# OS specific support (must be 'true' or 'false').
cygwin=false
msys=false
darwin=false
nonstop=false
-case "$( uname )" in #(
- CYGWIN* ) cygwin=true ;; #(
- Darwin* ) darwin=true ;; #(
- MSYS* | MINGW* ) msys=true ;; #(
- NONSTOP* ) nonstop=true ;;
+case "`uname`" in
+ CYGWIN* )
+ cygwin=true
+ ;;
+ Darwin* )
+ darwin=true
+ ;;
+ MINGW* )
+ msys=true
+ ;;
+ NONSTOP* )
+ nonstop=true
+ ;;
esac
CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
@@ -121,9 +87,9 @@ CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
if [ -n "$JAVA_HOME" ] ; then
if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
# IBM's JDK on AIX uses strange locations for the executables
- JAVACMD=$JAVA_HOME/jre/sh/java
+ JAVACMD="$JAVA_HOME/jre/sh/java"
else
- JAVACMD=$JAVA_HOME/bin/java
+ JAVACMD="$JAVA_HOME/bin/java"
fi
if [ ! -x "$JAVACMD" ] ; then
die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
@@ -132,7 +98,7 @@ Please set the JAVA_HOME variable in your environment to match the
location of your Java installation."
fi
else
- JAVACMD=java
+ JAVACMD="java"
which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
Please set the JAVA_HOME variable in your environment to match the
@@ -140,95 +106,80 @@ location of your Java installation."
fi
# Increase the maximum file descriptors if we can.
-if ! "$cygwin" && ! "$darwin" && ! "$nonstop" ; then
- case $MAX_FD in #(
- max*)
- MAX_FD=$( ulimit -H -n ) ||
- warn "Could not query maximum file descriptor limit"
- esac
- case $MAX_FD in #(
- '' | soft) :;; #(
- *)
- ulimit -n "$MAX_FD" ||
- warn "Could not set maximum file descriptor limit to $MAX_FD"
- esac
+if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
+ MAX_FD_LIMIT=`ulimit -H -n`
+ if [ $? -eq 0 ] ; then
+ if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
+ MAX_FD="$MAX_FD_LIMIT"
+ fi
+ ulimit -n $MAX_FD
+ if [ $? -ne 0 ] ; then
+ warn "Could not set maximum file descriptor limit: $MAX_FD"
+ fi
+ else
+ warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
+ fi
fi
-# Collect all arguments for the java command, stacking in reverse order:
-# * args from the command line
-# * the main class name
-# * -classpath
-# * -D...appname settings
-# * --module-path (only if needed)
-# * DEFAULT_JVM_OPTS, JAVA_OPTS, and GRADLE_OPTS environment variables.
+# For Darwin, add options to specify how the application appears in the dock
+if $darwin; then
+ GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
+fi
# For Cygwin or MSYS, switch paths to Windows format before running java
-if "$cygwin" || "$msys" ; then
- APP_HOME=$( cygpath --path --mixed "$APP_HOME" )
- CLASSPATH=$( cygpath --path --mixed "$CLASSPATH" )
-
- JAVACMD=$( cygpath --unix "$JAVACMD" )
-
+if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
+ APP_HOME=`cygpath --path --mixed "$APP_HOME"`
+ CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
+
+ JAVACMD=`cygpath --unix "$JAVACMD"`
+
+ # We build the pattern for arguments to be converted via cygpath
+ ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
+ SEP=""
+ for dir in $ROOTDIRSRAW ; do
+ ROOTDIRS="$ROOTDIRS$SEP$dir"
+ SEP="|"
+ done
+ OURCYGPATTERN="(^($ROOTDIRS))"
+ # Add a user-defined pattern to the cygpath arguments
+ if [ "$GRADLE_CYGPATTERN" != "" ] ; then
+ OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
+ fi
# Now convert the arguments - kludge to limit ourselves to /bin/sh
- for arg do
- if
- case $arg in #(
- -*) false ;; # don't mess with options #(
- /?*) t=${arg#/} t=/${t%%/*} # looks like a POSIX filepath
- [ -e "$t" ] ;; #(
- *) false ;;
- esac
- then
- arg=$( cygpath --path --ignore --mixed "$arg" )
+ i=0
+ for arg in "$@" ; do
+ CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
+ CHECK2=`echo "$arg"|egrep -c "^-"` ### Determine if an option
+
+ if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then ### Added a condition
+ eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
+ else
+ eval `echo args$i`="\"$arg\""
fi
- # Roll the args list around exactly as many times as the number of
- # args, so each arg winds up back in the position where it started, but
- # possibly modified.
- #
- # NB: a `for` loop captures its iteration list before it begins, so
- # changing the positional parameters here affects neither the number of
- # iterations, nor the values presented in `arg`.
- shift # remove old arg
- set -- "$@" "$arg" # push replacement arg
+ i=`expr $i + 1`
done
+ case $i in
+ 0) set -- ;;
+ 1) set -- "$args0" ;;
+ 2) set -- "$args0" "$args1" ;;
+ 3) set -- "$args0" "$args1" "$args2" ;;
+ 4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+ 5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+ 6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+ 7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+ 8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+ 9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+ esac
fi
-# Collect all arguments for the java command;
-# * $DEFAULT_JVM_OPTS, $JAVA_OPTS, and $GRADLE_OPTS can contain fragments of
-# shell script including quotes and variable substitutions, so put them in
-# double quotes to make sure that they get re-expanded; and
-# * put everything else in single quotes, so that it's not re-expanded.
-
-set -- \
- "-Dorg.gradle.appname=$APP_BASE_NAME" \
- -classpath "$CLASSPATH" \
- org.gradle.wrapper.GradleWrapperMain \
- "$@"
-
-# Use "xargs" to parse quoted args.
-#
-# With -n1 it outputs one arg per line, with the quotes and backslashes removed.
-#
-# In Bash we could simply go:
-#
-# readarray ARGS < <( xargs -n1 <<<"$var" ) &&
-# set -- "${ARGS[@]}" "$@"
-#
-# but POSIX shell has neither arrays nor command substitution, so instead we
-# post-process each arg (as a line of input to sed) to backslash-escape any
-# character that might be a shell metacharacter, then use eval to reverse
-# that process (while maintaining the separation between arguments), and wrap
-# the whole thing up as a single "set" statement.
-#
-# This will of course break if any of these variables contains a newline or
-# an unmatched quote.
-#
+# Escape application args
+save () {
+ for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
+ echo " "
+}
+APP_ARGS=`save "$@"`
-eval "set -- $(
- printf '%s\n' "$DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS" |
- xargs -n1 |
- sed ' s~[^-[:alnum:]+,./:=@_]~\\&~g; ' |
- tr '\n' ' '
- )" '"$@"'
+# Collect all arguments for the java command, following the shell quoting and substitution rules
+eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
exec "$JAVACMD" "$@"
diff --git a/src/main/java/com/kustacks/kuring/notice/domain/Notice.java b/src/main/java/com/kustacks/kuring/notice/domain/Notice.java
index 13786e49..00a58b2e 100644
--- a/src/main/java/com/kustacks/kuring/notice/domain/Notice.java
+++ b/src/main/java/com/kustacks/kuring/notice/domain/Notice.java
@@ -25,6 +25,7 @@ public class Notice {
@Column(name = "posted_dt", length = 32, nullable = false)
private String postedDate;
+ @Getter(AccessLevel.PUBLIC)
@Column(name = "updated_dt", length = 32)
private String updatedDate;
@@ -55,6 +56,9 @@ public Notice(String articleId, String postedDate, String updatedDate,
this.url = new Url(fullUrl);
}
+ public boolean isImportant() {
+ return this.important;
+ }
public String getCategoryName() {
return this.categoryName.getName();
}
diff --git a/src/main/java/com/kustacks/kuring/notice/domain/NoticeJdbcRepository.java b/src/main/java/com/kustacks/kuring/notice/domain/NoticeJdbcRepository.java
new file mode 100644
index 00000000..4da21053
--- /dev/null
+++ b/src/main/java/com/kustacks/kuring/notice/domain/NoticeJdbcRepository.java
@@ -0,0 +1,64 @@
+package com.kustacks.kuring.notice.domain;
+
+import lombok.RequiredArgsConstructor;
+import org.springframework.jdbc.core.BatchPreparedStatementSetter;
+import org.springframework.jdbc.core.JdbcTemplate;
+import org.springframework.stereotype.Repository;
+import org.springframework.transaction.annotation.Transactional;
+
+import java.sql.PreparedStatement;
+import java.sql.SQLException;
+import java.util.List;
+
+@Repository
+@RequiredArgsConstructor
+public class NoticeJdbcRepository {
+
+ private final JdbcTemplate jdbcTemplate;
+
+ @Transactional
+ public void saveAllCategoryNotices(List notices) {
+ jdbcTemplate.batchUpdate("INSERT INTO notice (article_id, category_name, important, posted_dt, subject, updated_dt, url, dtype) values (?, ?, ?, ?, ?, ?, ?, 'Notice')",
+ new BatchPreparedStatementSetter() {
+ @Override
+ public void setValues(PreparedStatement ps, int i) throws SQLException {
+ Notice notice = notices.get(i);
+ ps.setString(1, notice.getArticleId());
+ ps.setString(2, notice.getCategoryName().toUpperCase());
+ ps.setInt(3, notice.isImportant() ? 1 : 0);
+ ps.setString(4, notice.getPostedDate());
+ ps.setString(5, notice.getSubject());
+ ps.setString(6, notice.getUpdatedDate());
+ ps.setString(7, notice.getUrl());
+ }
+
+ @Override
+ public int getBatchSize() {
+ return notices.size();
+ }
+ });
+ }
+
+ @Transactional
+ public void saveAllDepartmentNotices(List departmentNotices) {
+ jdbcTemplate.batchUpdate("INSERT INTO notice (article_id, category_name, important, posted_dt, subject, updated_dt, url, dtype) values (?, ?, ?, ?, ?, ?, ?, 'DepartmentNotice')",
+ new BatchPreparedStatementSetter() {
+ @Override
+ public void setValues(PreparedStatement ps, int i) throws SQLException {
+ DepartmentNotice departmentNotice = departmentNotices.get(i);
+ ps.setString(1, departmentNotice.getArticleId());
+ ps.setString(2, departmentNotice.getCategoryName().toUpperCase());
+ ps.setInt(3, departmentNotice.isImportant() ? 1 : 0);
+ ps.setString(4, departmentNotice.getPostedDate());
+ ps.setString(5, departmentNotice.getSubject());
+ ps.setString(6, departmentNotice.getUpdatedDate());
+ ps.setString(7, departmentNotice.getUrl());
+ }
+
+ @Override
+ public int getBatchSize() {
+ return departmentNotices.size();
+ }
+ });
+ }
+}
diff --git a/src/main/java/com/kustacks/kuring/worker/update/notice/CategoryNoticeUpdater.java b/src/main/java/com/kustacks/kuring/worker/update/notice/CategoryNoticeUpdater.java
index dfd6b143..bc5c48c0 100644
--- a/src/main/java/com/kustacks/kuring/worker/update/notice/CategoryNoticeUpdater.java
+++ b/src/main/java/com/kustacks/kuring/worker/update/notice/CategoryNoticeUpdater.java
@@ -3,6 +3,7 @@
import com.kustacks.kuring.notice.domain.CategoryName;
import com.kustacks.kuring.message.firebase.FirebaseService;
import com.kustacks.kuring.notice.domain.Notice;
+import com.kustacks.kuring.notice.domain.NoticeJdbcRepository;
import com.kustacks.kuring.notice.domain.NoticeRepository;
import com.kustacks.kuring.worker.scrap.KuisNoticeScraperTemplate;
import com.kustacks.kuring.worker.scrap.client.notice.LibraryNoticeApiClient;
@@ -30,6 +31,7 @@ public class CategoryNoticeUpdater {
private final List kuisNoticeInfoList;
private final KuisNoticeScraperTemplate scrapperTemplate;
private final NoticeRepository noticeRepository;
+ private final NoticeJdbcRepository noticeJdbcRepository;
private final FirebaseService firebaseService;
private final LibraryNoticeApiClient libraryNoticeApiClient;
private final ThreadPoolTaskExecutor noticeUpdaterThreadTaskExecutor;
@@ -61,15 +63,11 @@ private void updateLibrary() {
}
private List updateLibraryNotice(CategoryName categoryName) {
- List scrapResults = libraryNoticeApiClient.request(categoryName);
- Collections.reverse(scrapResults);
- return scrapResults;
+ return libraryNoticeApiClient.request(categoryName);
}
private List updateKuisNoticeAsync(KuisNoticeInfo deptInfo, Function> decisionMaker) {
- List scrapResults = scrapperTemplate.scrap(deptInfo, decisionMaker);
- Collections.reverse(scrapResults);
- return scrapResults;
+ return scrapperTemplate.scrap(deptInfo, decisionMaker);
}
private List compareLatestAndUpdateDB(List scrapResults, CategoryName categoryName) {
@@ -92,7 +90,7 @@ private List synchronizationWithDb(List scrapResu
List deletedNoticesArticleIds = filteringSoonDeleteIds(savedArticleIds, scrapNoticeIds);
- noticeRepository.saveAllAndFlush(newNotices);
+ noticeJdbcRepository.saveAllCategoryNotices(newNotices);
if (!deletedNoticesArticleIds.isEmpty()) {
noticeRepository.deleteAllByIdsAndCategory(categoryName, deletedNoticesArticleIds);
diff --git a/src/main/java/com/kustacks/kuring/worker/update/notice/DepartmentNoticeUpdater.java b/src/main/java/com/kustacks/kuring/worker/update/notice/DepartmentNoticeUpdater.java
index e75a0455..dc54e034 100644
--- a/src/main/java/com/kustacks/kuring/worker/update/notice/DepartmentNoticeUpdater.java
+++ b/src/main/java/com/kustacks/kuring/worker/update/notice/DepartmentNoticeUpdater.java
@@ -1,10 +1,12 @@
package com.kustacks.kuring.worker.update.notice;
-import com.kustacks.kuring.notice.domain.CategoryName;
+
import com.kustacks.kuring.message.firebase.FirebaseService;
+import com.kustacks.kuring.notice.domain.CategoryName;
import com.kustacks.kuring.notice.domain.DepartmentName;
import com.kustacks.kuring.notice.domain.DepartmentNotice;
import com.kustacks.kuring.notice.domain.DepartmentNoticeRepository;
+import com.kustacks.kuring.notice.domain.NoticeJdbcRepository;
import com.kustacks.kuring.worker.scrap.DepartmentNoticeScraperTemplate;
import com.kustacks.kuring.worker.scrap.deptinfo.DeptInfo;
import com.kustacks.kuring.worker.scrap.dto.ComplexNoticeFormatDto;
@@ -34,6 +36,7 @@ public class DepartmentNoticeUpdater {
private final List deptInfoList;
private final DepartmentNoticeScraperTemplate scrapperTemplate;
+ private final NoticeJdbcRepository noticeJdbcRepository;
private final DepartmentNoticeRepository departmentNoticeRepository;
private final ThreadPoolTaskExecutor noticeUpdaterThreadTaskExecutor;
private final FirebaseService firebaseService;
@@ -107,7 +110,7 @@ private List compareLatestAndUpdateDB(List saveNewNotices(List scrapResults, List savedArticleIds, DepartmentName departmentNameEnum, boolean important) {
List newNotices = filteringSoonSaveNotice(scrapResults, savedArticleIds, departmentNameEnum, important);
- departmentNoticeRepository.saveAllAndFlush(newNotices);
+ noticeJdbcRepository.saveAllDepartmentNotices(newNotices);
return newNotices;
}
@@ -143,7 +146,7 @@ private void synchronizationWithDb(List scrapResults, Lis
List deletedNoticesArticleIds = filteringSoonDeleteIds(savedArticleIds, latestNoticeIds);
- departmentNoticeRepository.saveAllAndFlush(newNotices);
+ noticeJdbcRepository.saveAllDepartmentNotices(newNotices);
if (!deletedNoticesArticleIds.isEmpty()) {
departmentNoticeRepository.deleteAllByIdsAndDepartment(departmentNameEnum, deletedNoticesArticleIds);
diff --git a/src/main/resources/application.yml b/src/main/resources/application.yml
index 7c72eea6..69e4fbf5 100644
--- a/src/main/resources/application.yml
+++ b/src/main/resources/application.yml
@@ -11,6 +11,8 @@ spring:
ddl-auto: none
properties:
hibernate:
+ jdbc:
+ batch_size: 100
format_sql: true
dialect: com.kustacks.kuring.config.CustomMariaDbDialect
defer-datasource-initialization: false
diff --git a/src/test/java/com/kustacks/kuring/worker/update/CategoryNoticeUpdaterTest.java b/src/test/java/com/kustacks/kuring/worker/update/CategoryNoticeUpdaterTest.java
new file mode 100644
index 00000000..c4f4afad
--- /dev/null
+++ b/src/test/java/com/kustacks/kuring/worker/update/CategoryNoticeUpdaterTest.java
@@ -0,0 +1,105 @@
+package com.kustacks.kuring.worker.update;
+
+import com.kustacks.kuring.message.firebase.FirebaseService;
+import com.kustacks.kuring.notice.domain.Notice;
+import com.kustacks.kuring.notice.domain.NoticeRepository;
+import com.kustacks.kuring.worker.scrap.KuisNoticeScraperTemplate;
+import com.kustacks.kuring.worker.scrap.client.notice.LibraryNoticeApiClient;
+import com.kustacks.kuring.worker.update.notice.CategoryNoticeUpdater;
+import com.kustacks.kuring.worker.update.notice.dto.response.CommonNoticeFormatDto;
+import org.assertj.core.api.Assertions;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.test.context.SpringBootTest;
+import org.springframework.boot.test.mock.mockito.MockBean;
+import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
+import org.springframework.test.context.TestPropertySource;
+
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import static org.junit.jupiter.api.Assertions.assertAll;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.doReturn;
+
+
+@SpringBootTest
+@TestPropertySource(properties = "spring.config.location=" +
+ "classpath:/application.yml" +
+ ",classpath:/application-test.yml" +
+ ",classpath:/test-constants.properties")
+class CategoryNoticeUpdaterTest {
+
+ @MockBean
+ KuisNoticeScraperTemplate scrapperTemplate;
+
+ @MockBean
+ FirebaseService firebaseService;
+
+ @MockBean
+ LibraryNoticeApiClient libraryNoticeApiClient;
+
+ @Autowired
+ CategoryNoticeUpdater categoryNoticeUpdater;
+
+ @Autowired
+ ThreadPoolTaskExecutor noticeUpdaterThreadTaskExecutor;
+
+ @Autowired
+ NoticeRepository noticeRepository;
+
+ @DisplayName("공지 업데이트 테스트")
+ @Test
+ void notice_scrap_async_test() throws InterruptedException {
+ // given
+ doReturn(createNoticesFixture()).when(scrapperTemplate).scrap(any(), any());
+ doReturn(createLibraryFixture()).when(libraryNoticeApiClient).request(any());
+ doNothing().when(firebaseService).sendNotificationList(anyList());
+
+ // when
+ categoryNoticeUpdater.update();
+ noticeUpdaterThreadTaskExecutor.getThreadPoolExecutor().awaitTermination(1, TimeUnit.SECONDS);
+ List notices = noticeRepository.findAll();
+
+ // then
+ assertAll(
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "library").size().isEqualTo(7),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "bachelor").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "scholarship").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "employment").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "national").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "student").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "industry_university").size().isEqualTo(9),
+ () -> Assertions.assertThat(notices).filteredOn("categoryName", "normal").size().isEqualTo(9)
+ );
+ }
+
+ private static List createLibraryFixture() {
+ return List.of(
+ new CommonNoticeFormatDto("1", "2021-01-01", "library1", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71921", false),
+ new CommonNoticeFormatDto("2", "2021-01-01", "library2", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71922", false),
+ new CommonNoticeFormatDto("3", "2021-01-01", "library3", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71923", false),
+ new CommonNoticeFormatDto("4", "2021-01-01", "library4", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71924", false),
+ new CommonNoticeFormatDto("5", "2021-01-01", "library5", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71925", false),
+ new CommonNoticeFormatDto("6", "2021-01-01", "library6", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71926", false),
+ new CommonNoticeFormatDto("7", "2021-01-01", "library7", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71927", false)
+ );
+ }
+
+ private static List createNoticesFixture() {
+ return List.of(
+ new CommonNoticeFormatDto("1", "2021-01-01", "subject1", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71921", false),
+ new CommonNoticeFormatDto("2", "2021-01-01", "subject2", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71922", false),
+ new CommonNoticeFormatDto("3", "2021-01-01", "subject3", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71923", false),
+ new CommonNoticeFormatDto("4", "2021-01-01", "subject4", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71924", false),
+ new CommonNoticeFormatDto("5", "2021-01-01", "subject5", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71925", false),
+ new CommonNoticeFormatDto("6", "2021-01-01", "subject6", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71926", false),
+ new CommonNoticeFormatDto("7", "2021-01-01", "subject7", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71927", false),
+ new CommonNoticeFormatDto("8", "2021-01-01", "subject8", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71928", false),
+ new CommonNoticeFormatDto("9", "2021-01-01", "subject9", "2021-01-01", "https://library.konkuk.ac.kr/library-guide/bulletins/notice/71929", false)
+ );
+ }
+}
diff --git a/src/test/java/com/kustacks/kuring/worker/update/DepartmentNoticeUpdaterTest.java b/src/test/java/com/kustacks/kuring/worker/update/DepartmentNoticeUpdaterTest.java
new file mode 100644
index 00000000..f5d99ecf
--- /dev/null
+++ b/src/test/java/com/kustacks/kuring/worker/update/DepartmentNoticeUpdaterTest.java
@@ -0,0 +1,88 @@
+package com.kustacks.kuring.worker.update;
+
+import com.kustacks.kuring.message.firebase.FirebaseService;
+import com.kustacks.kuring.notice.domain.DepartmentNotice;
+import com.kustacks.kuring.notice.domain.DepartmentNoticeRepository;
+import com.kustacks.kuring.worker.scrap.DepartmentNoticeScraperTemplate;
+import com.kustacks.kuring.worker.scrap.dto.ComplexNoticeFormatDto;
+import com.kustacks.kuring.worker.update.notice.DepartmentNoticeUpdater;
+import com.kustacks.kuring.worker.update.notice.dto.response.CommonNoticeFormatDto;
+import org.junit.jupiter.api.DisplayName;
+import org.junit.jupiter.api.Test;
+import org.springframework.beans.factory.annotation.Autowired;
+import org.springframework.boot.test.context.SpringBootTest;
+import org.springframework.boot.test.mock.mockito.MockBean;
+import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
+import org.springframework.test.context.TestPropertySource;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.TimeUnit;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.ArgumentMatchers.anyList;
+import static org.mockito.Mockito.doNothing;
+import static org.mockito.Mockito.doReturn;
+
+
+@SpringBootTest
+@TestPropertySource(properties = "spring.config.location=" +
+ "classpath:/application.yml" +
+ ",classpath:/application-test.yml" +
+ ",classpath:/test-constants.properties")
+class DepartmentNoticeUpdaterTest {
+
+ @MockBean
+ DepartmentNoticeScraperTemplate scrapperTemplate;
+
+ @MockBean
+ FirebaseService firebaseService;
+
+ @Autowired
+ DepartmentNoticeUpdater departmentNoticeUpdater;
+
+ @Autowired
+ ThreadPoolTaskExecutor noticeUpdaterThreadTaskExecutor;
+
+ @Autowired
+ DepartmentNoticeRepository departmentNoticeRepository;
+
+ @DisplayName("학과별 공지 업데이트 테스트")
+ @Test
+ void department_scrap_async_test() throws InterruptedException {
+ // given
+ doReturn(createDepartmentNoticesFixture()).when(scrapperTemplate).scrap(any(), any());
+ doNothing().when(firebaseService).sendNotificationList(anyList());
+
+ // when
+ departmentNoticeUpdater.update();
+ noticeUpdaterThreadTaskExecutor.getThreadPoolExecutor().awaitTermination(2, TimeUnit.SECONDS);
+ List notices = departmentNoticeRepository.findAll();
+
+
+ // then
+ assertThat(notices).hasSize(3720);
+ }
+
+ private static List createDepartmentNoticesFixture() {
+ List result = new ArrayList<>();
+ List importantNoticeList = new ArrayList<>();
+ List normalNoticeList = new ArrayList<>();
+
+ for(int i = 0; i < 30; i++) {
+ importantNoticeList.add(new CommonNoticeFormatDto(String.valueOf(i), "2021-01-01",
+ "important" + i, "2021-01-01",
+ "https://library.konkuk.ac.kr/library-guide/bulletins/important/71921",
+ true));
+
+ normalNoticeList.add(new CommonNoticeFormatDto(String.valueOf(i), "2021-01-01",
+ "normal" + i, "2021-01-01",
+ "https://library.konkuk.ac.kr/library-guide/bulletins/normal/71921",
+ false));
+ }
+
+ result.add(new ComplexNoticeFormatDto(importantNoticeList, normalNoticeList));
+ return result;
+ }
+}
diff --git a/src/test/resources/application-test.yml b/src/test/resources/application-test.yml
new file mode 100644
index 00000000..95666fbb
--- /dev/null
+++ b/src/test/resources/application-test.yml
@@ -0,0 +1,49 @@
+spring:
+ datasource:
+ url: jdbc:tc:mariadb:10.2:///kuring?TC_REUSABLE=true
+ username: test # 테스트컨테이너 용이라 노출되도 상관없다
+ password: test1234
+ driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
+ p6spy:
+ enable-logging: true
+ hikari:
+ jdbc-url: jdbc:tc:mariadb:10.2://localhost:3306/kuring?profileSQL=true&logger=Slf4JLogger&maxQuerySizeToLog=999999
+ flyway:
+ enabled: false
+ jpa:
+ hibernate:
+ ddl-auto: create
+ show-sql: true
+ properties:
+ hibernate:
+ jdbc:
+ batch_size: 100
+ format_sql: true
+ dialect: com.kustacks.kuring.config.CustomMariaDbDialect
+ defer-datasource-initialization: true
+ open-in-view: false
+ thymeleaf:
+ cache: true
+ enabled: true
+ prefix: classpath:/templates/
+ suffix: .html
+ jackson:
+ serialization:
+ FAIL_ON_EMPTY_BEANS: false
+
+testcontainers:
+ reuse:
+ enable: true
+
+sentry:
+ dsn: ""
+
+logging:
+ level:
+ p6spy: debug
+
+security:
+ jwt:
+ token:
+ secret-key: test-secret-key-test-secret-key-test-secret-key-test-secret-key
+ expire-length: 3600000
diff --git a/src/test/resources/application.yml b/src/test/resources/application.yml
index f5f3a650..03c30d37 100644
--- a/src/test/resources/application.yml
+++ b/src/test/resources/application.yml
@@ -1,45 +1,3 @@
spring:
- datasource:
- url: jdbc:tc:mariadb:10.2:///kuring?TC_REUSABLE=true
- username: test # 테스트컨테이너 용이라 노출되도 상관없다
- password: test1234
- driver-class-name: org.testcontainers.jdbc.ContainerDatabaseDriver
- p6spy:
- enable-logging: true
- flyway:
- enabled: false
- jpa:
- hibernate:
- ddl-auto: create
- show-sql: true
- properties:
- hibernate:
- format_sql: true
- dialect: com.kustacks.kuring.config.CustomMariaDbDialect
- defer-datasource-initialization: true
- open-in-view: false
- thymeleaf:
- cache: true
- enabled: true
- prefix: classpath:/templates/
- suffix: .html
- jackson:
- serialization:
- FAIL_ON_EMPTY_BEANS: false
-
-testcontainers:
- reuse:
- enable: true
-
-sentry:
- dsn: ""
-
-logging:
- level:
- p6spy: debug
-
-security:
- jwt:
- token:
- secret-key: test-secret-key-test-secret-key-test-secret-key-test-secret-key
- expire-length: 3600000
+ profiles:
+ active: test
diff --git a/system.properties b/system.properties
index 180a2734..eafd676c 100644
--- a/system.properties
+++ b/system.properties
@@ -1 +1 @@
-java.runtime.version=11
\ No newline at end of file
+java.runtime.version=17
