Development/Spring Boot3 (Kotlin)
[Kotlin][SpringBoot3] Kopring 서버 기본 실습 18 - 외부 api를 활용해 Scheduler로 정보 수집 실습 5 : yahoo finance에서 주요 환율 정보 크롤링해서 저장하기
Tradgineer
2023. 10. 16. 15:47
1. 이전 포스팅
https://growingsaja.tistory.com/989
2. 목표
- 야후 파이낸스에서 주요 환율 정보 크롤링 및 가공해서 저장하기
https://finance.yahoo.com/quote/KRW=X?p=KRW=X&.tsrc=fin-srch
fin-streamer 안에 주요 데이터들이 있습니다.
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="ES=F" data-field="regularMarketPrice" data-trend="none" value="4331.5" active="true">
4,331.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="ES=F" data-field="regularMarketChange" data-trend="txt" value="16.75" active="true">
<span class="C($positiveColor)">+16.75</span>
</fin-streamer>
<fin-streamer data-symbol="ES=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.38820326" active="true">
<span class="C($positiveColor)">(+0.39%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="YM=F" data-field="regularMarketPrice" data-trend="none" value="33984" active="true">
33,984.00
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="YM=F" data-field="regularMarketChange" data-trend="txt" value="109" active="true">
<span class="C($positiveColor)">+109.00</span>
</fin-streamer>
<fin-streamer data-symbol="YM=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.32177123" active="true">
<span class="C($positiveColor)">(+0.32%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="NQ=F" data-field="regularMarketPrice" data-trend="none" value="14770.5" active="true">
14,770.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="NQ=F" data-field="regularMarketChange" data-trend="txt" value="55.5" active="true">
<span class="C($positiveColor)">+55.50</span>
</fin-streamer>
<fin-streamer data-symbol="NQ=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.37716615" active="true">
<span class="C($positiveColor)">(+0.38%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="RTY=F" data-field="regularMarketPrice" data-trend="none" value="1786.5" active="true">
1,786.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="RTY=F" data-field="regularMarketChange" data-trend="txt" value="8.599976" active="true">
<span class="C($positiveColor)">+8.60</span>
</fin-streamer>
<fin-streamer data-symbol="RTY=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.48371536" active="true">
<span class="C($positiveColor)">(+0.48%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="CL=F" data-field="regularMarketPrice" data-trend="none" value="91.15" active="true">
91.15
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="CL=F" data-field="regularMarketChange" data-trend="txt" value="0.76000214" active="true">
<span class="C($positiveColor)">+0.76</span>
</fin-streamer>
<fin-streamer data-symbol="CL=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.8408033" active="true">
<span class="C($positiveColor)">(+0.84%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="GC=F" data-field="regularMarketPrice" data-trend="none" value="1915.1" active="true">
1,915.10
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="GC=F" data-field="regularMarketChange" data-trend="txt" value="-4.7000732" active="true">
<span class="C($negativeColor)">-4.70</span>
</fin-streamer>
<fin-streamer data-symbol="GC=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="-0.24482098" active="true">
<span class="C($negativeColor)">(-0.24%)</span>
</fin-streamer>
<fin-streamer class="Fw(b) Fz(36px) Mb(-4px) D(ib)" data-symbol="KRW=X" data-test="qsp-price" data-field="regularMarketPrice" data-trend="none" data-pricehint="4" value="1352.74" active="">
1,352.7400
</fin-streamer>
<fin-streamer class="Fw(500) Pstart(8px) Fz(24px)" data-symbol="KRW=X" data-test="qsp-price-change" data-field="regularMarketChange" data-trend="txt" data-pricehint="4" value="0.10998535" active="">
<span class="C($positiveColor)">+0.1100</span>
</fin-streamer>
<fin-streamer class="Fw(500) Pstart(8px) Fz(24px)" data-symbol="KRW=X" data-field="regularMarketChangePercent" data-trend="txt" data-pricehint="4" data-template="({fmt})" value="0.00008131223" active="">
<span class="C($positiveColor)">(+0.0081%)</span>
</fin-streamer>
<fin-streamer class="D(n)" data-symbol="KRW=X" changeev="regularTimeChange" data-field="regularMarketTime" data-trend="none" value="" active="true"></fin-streamer>
<fin-streamer class="D(n)" data-symbol="KRW=X" changeev="marketState" data-field="marketState" data-trend="none" value="" active="true"></fin-streamer>
1. USD/KRW 환율 가격: 4,331.50 +16.75 (+0.39%) 33,984.00 +109.00 (+0.32%) 14,770.50 +55.50 (+0.38%) 1,786.50 +8.60 (+0.48%) 91.15 +0.76 (+0.84%) 1,915.10 -4.70 (-0.24%) 1,352.7400 +0.1100 (+0.0081%)
2023-09-27T16:43:26.893+09:00 INFO 3273 --- [ scheduling-1] c.d.k.e.f.s.YahooFinanceRateScheduler : [ ExternalApi:OpenExchangeRate ] Success get Data from api
2023-09-27T16:43:26.894+09:00 INFO 3273 --- [ scheduling-1] c.d.k.e.f.s.YahooFinanceRateScheduler : [ ExternalApi:Yahoo Finance Rate ] Start get Data from web
2023-09-27T16:43:27.750+09:00 INFO 3273 --- [ionShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="ES=F" data-field="regularMarketPrice" data-trend="none" value="4331.5" active="true">
4,331.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="ES=F" data-field="regularMarketChange" data-trend="txt" value="16.75" active="true">
<span class="C($positiveColor)">+16.75</span>
</fin-streamer>
<fin-streamer data-symbol="ES=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.38820326" active="true">
<span class="C($positiveColor)">(+0.39%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="YM=F" data-field="regularMarketPrice" data-trend="none" value="33983" active="true">
33,983.00
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="YM=F" data-field="regularMarketChange" data-trend="txt" value="108" active="true">
<span class="C($positiveColor)">+108.00</span>
</fin-streamer>
<fin-streamer data-symbol="YM=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.3188192" active="true">
<span class="C($positiveColor)">(+0.32%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="NQ=F" data-field="regularMarketPrice" data-trend="none" value="14770.5" active="true">
14,770.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="NQ=F" data-field="regularMarketChange" data-trend="txt" value="55.5" active="true">
<span class="C($positiveColor)">+55.50</span>
</fin-streamer>
<fin-streamer data-symbol="NQ=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.37716615" active="true">
<span class="C($positiveColor)">(+0.38%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="RTY=F" data-field="regularMarketPrice" data-trend="none" value="1786.5" active="true">
1,786.50
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="RTY=F" data-field="regularMarketChange" data-trend="txt" value="8.599976" active="true">
<span class="C($positiveColor)">+8.60</span>
</fin-streamer>
<fin-streamer data-symbol="RTY=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.48371536" active="true">
<span class="C($positiveColor)">(+0.48%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="CL=F" data-field="regularMarketPrice" data-trend="none" value="91.15" active="true">
91.15
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="CL=F" data-field="regularMarketChange" data-trend="txt" value="0.76000214" active="true">
<span class="C($positiveColor)">+0.76</span>
</fin-streamer>
<fin-streamer data-symbol="CL=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="0.8408033" active="true">
<span class="C($positiveColor)">(+0.84%)</span>
</fin-streamer>
<fin-streamer class="Fz(s) Mt(4px) Mb(0px) Fw(b) D(ib)" data-symbol="GC=F" data-field="regularMarketPrice" data-trend="none" value="1915" active="true">
1,915.00
</fin-streamer>
<fin-streamer class="Mend(2px)" data-symbol="GC=F" data-field="regularMarketChange" data-trend="txt" value="-4.800049" active="true">
<span class="C($negativeColor)">-4.80</span>
</fin-streamer>
<fin-streamer data-symbol="GC=F" data-field="regularMarketChangePercent" data-trend="txt" data-template="({fmt})" value="-0.25002858" active="true">
<span class="C($negativeColor)">(-0.25%)</span>
</fin-streamer>
<fin-streamer class="Fw(b) Fz(36px) Mb(-4px) D(ib)" data-symbol="KRW=X" data-test="qsp-price" data-field="regularMarketPrice" data-trend="none" data-pricehint="4" value="1352.78" active="">
1,352.7800
</fin-streamer>
<fin-streamer class="Fw(500) Pstart(8px) Fz(24px)" data-symbol="KRW=X" data-test="qsp-price-change" data-field="regularMarketChange" data-trend="txt" data-pricehint="4" value="0.15002441" active="">
<span class="C($positiveColor)">+0.1500</span>
</fin-streamer>
<fin-streamer class="Fw(500) Pstart(8px) Fz(24px)" data-symbol="KRW=X" data-field="regularMarketChangePercent" data-trend="txt" data-pricehint="4" data-template="({fmt})" value="0.00011091311" active="">
<span class="C($positiveColor)">(+0.0111%)</span>
</fin-streamer>
<fin-streamer class="D(n)" data-symbol="KRW=X" changeev="regularTimeChange" data-field="regularMarketTime" data-trend="none" value="" active="true"></fin-streamer>
<fin-streamer class="D(n)" data-symbol="KRW=X" changeev="marketState" data-field="marketState" data-trend="none" value="" active="true"></fin-streamer>
3. 웹 크롤링 및 데이터 가공 추출 예제 소스코드 2가지
- 방법 1
// ...
import org.jsoup.Jsoup
// ...
fun getDataFromWeb(apiUrl: String) {
try {
val document = Jsoup.connect(apiUrl).get()
// 1. usdkrw 환율 가격 실시간 정보
val finData = document.select("fin-streamer")
val exchangeRateText = finData.select("[data-symbol='KRW=X'][data-field='regularMarketPrice']").first()?.text()
println("1. USD/KRW 환율 가격: $exchangeRateText")
} catch (e: Exception) {
// 오류 처리
println("환율 정보를 가져오는 데 실패했습니다: ${e.message}")
}
}
- 방법 2
// ...
import org.jsoup.Jsoup
// ...
fun getDataFromWeb(apiUrl: String) {
try {
val document = Jsoup.connect(apiUrl).get()
// 1. usdkrw 환율 가격 실시간 정보
val exchangeRateText = document.select("fin-streamer[data-symbol='KRW=X'][data-field='regularMarketPrice']").first()?.text()
println("1. USD/KRW 환율 가격: $exchangeRateText")
} catch (e: Exception) {
// 오류 처리
println("환율 정보를 가져오는 데 실패했습니다: ${e.message}")
}
}
4. 프로젝트 구조 및 소스코드 구성
5. entity, repository 소스코드 작성
// vim YahooRateData.kt
package com.dev.kopring00.external.fiat.entities
import org.springframework.data.mongodb.core.index.Indexed
import org.springframework.data.mongodb.core.mapping.Document
import java.time.Instant
@Document(collection = "yahooRate")
data class YahooRateData(
@Indexed
val code: String,
val price: Double?,
val changePrice: Double?,
val changeRate: Float?,
val bid: Double?,
val ask: Double?,
val lowPrice: Double?,
val highPrice: Double?,
val low52wPrice: Double?,
val high52wPrice: Double?,
val timestamp: Long = Instant.now().epochSecond
)
// vim YahooMainData.kt
package com.dev.kopring00.external.fiat.entities
import org.springframework.data.mongodb.core.index.Indexed
import org.springframework.data.mongodb.core.mapping.Document
import java.time.Instant
@Document(collection = "yahooMain")
data class YahooMainData(
@Indexed
val name: String,
val price: Float?,
val change: Float?,
val percent: Float?,
val timestamp: Long = Instant.now().epochSecond
)
// vim YahooUsdkrwAndMainData.kt
package com.dev.kopring00.external.fiat.entities
data class YahooUsdkrwAndMainData(
val usdkrw: YahooRateData?,
val mainData: List<YahooMainData>?,
val isSuccess: Boolean
)
6. service, scheduler 소스코드 작성
// vim YahooRateService.kt
package com.dev.kopring00.external.fiat.services
import com.dev.kopring00.external.fiat.entities.YahooMainData
import com.dev.kopring00.external.fiat.entities.YahooRateData
import com.dev.kopring00.external.fiat.entities.YahooUsdkrwAndMainData
import com.dev.kopring00.external.fiat.repositories.YahooMainRepository
import org.jsoup.Jsoup
import com.dev.kopring00.external.fiat.repositories.YahooRateRepository
import org.jsoup.nodes.Element
import org.jsoup.select.Elements
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.springframework.stereotype.Service
@Service
class YahooRateService(
private val yahooRateRepository: YahooRateRepository,
private val yahooMainRepository: YahooMainRepository
) {
fun getMainDataWithUsdkrwFromWeb(apiUrl: String): YahooUsdkrwAndMainData {
val logger: Logger = LoggerFactory.getLogger(YahooRateService::class.java)
try {
/* ===== 해당 화면 상단 개요 정보들 main data ===== */
// https://finance.yahoo.com/quote/KRW=X?p=KRW=X&.tsrc=fin-srch
val document = Jsoup.connect(apiUrl).get()
val finData = document.select("fin-streamer")
val mainDataList: List<YahooMainData> = setMainData(finData)
val rateData: YahooRateData = setRateData("USDKRW", "KRW=X", finData, document)
return YahooUsdkrwAndMainData(rateData, mainDataList, true)
} catch (e: Exception) {
// 오류 처리
logger.error("[ ExternalApi:Yahoo ] getMainDataWithUsdkrwFromWeb failed : ${e.message}")
return YahooUsdkrwAndMainData(null, null, false)
}
}
fun setMainData(finData: Elements): List<YahooMainData> {
val itemMap = mapOf(
"S&P500 Futures" to "ES=F", // 1번
"Dow Futures" to "YM=F",
"Nasdaq Futures" to "NQ=F",
"Russell 2000 Futures" to "RTY=F",
"Crude Oil" to "CL=F",
"Gold" to "GC=F",
"Silver" to "SI",
"EUR/USD" to "EURUSD"
)
val mainDataList = mutableListOf<YahooMainData>()
for ((name, symbol) in itemMap) {
val itemData = finData.select("[data-symbol='$symbol']")
val price: String? = itemData.select("[data-field='regularMarketPrice']").first()?.text() // 가격 = data-field : regularMarketPrice
val change: String? = itemData.select("[data-field='regularMarketChange']").first()?.text() // 변동가 = data-field : regularMarketChange
var percent: String? = itemData.select("[data-field='regularMarketChangePercent']").first()?.text() // 변동률 = data-field : regularMarketChangePercent
percent = percent?.split("%")?.first()
percent = percent?.split("(")?.last()
val mainData = YahooMainData(
name,
price?.replace(",", "")?.toFloat(),
change?.replace(",", "")?.toFloat(),
percent?.replace(",", "")?.toFloat(),
)
mainDataList.add(mainData)
}
return mainDataList
}
fun setRateData(code: String, symbol: String, finData: Elements, document: Element): YahooRateData {
val usdkrwData = finData.select("[data-symbol='$symbol']")
/* ===== usdkrw ===== */
// 가격
val price_tmp = usdkrwData.select("[data-field='regularMarketPrice']").first()?.text()
val price = price_tmp?.replace(",", "")
// 변동가
val changePrice_tmp = usdkrwData.select("[data-field='regularMarketChange']").first()?.text()
val changePrice = changePrice_tmp?.replace(",", "")
// 변동률
val changePercent_tmp = usdkrwData.select("[data-field='regularMarketChangePercent']").first()?.text()
?.split("%")
?.first()?.split("(")?.last()
val changePercent = changePercent_tmp?.replace(",", "")
// // 전일 종가 -> 저장 불필요하여 미진행 : 2023-10-04 cjy
// val prevClose_tmp = document.select("[data-test='PREV_CLOSE-value']").first()?.text()
// val prevClose = prevClose_tmp?.replace(",", "")
// 구매 주문 호가
val bid_tmp = document.select("[data-test='BID-value']").first()?.text()
val bid = bid_tmp?.replace(",", "")
// 판매 주문 호가
val ask_tmp = document.select("[data-test='ASK-value']").first()?.text()
val ask = ask_tmp?.replace(",", "")
val daysRange = document.select("[data-test='DAYS_RANGE-value']").first()?.text()
val lowPrice_tmp = daysRange?.split(" - ")?.first()
val lowPrice = lowPrice_tmp?.replace(",", "")
val highPrice_tmp = daysRange?.split(" - ")?.last()
val highPrice = highPrice_tmp?.replace(",", "")
val usdkrw_52weeksRange = document.select("[data-test='FIFTY_TWO_WK_RANGE-value']").first()?.text()
val low52wPrice_tmp = usdkrw_52weeksRange?.split(" - ")?.first()
val high52wPrice_tmp = usdkrw_52weeksRange?.split(" - ")?.last()
val low52wPrice = low52wPrice_tmp?.replace(",", "")
val high52wPrice = high52wPrice_tmp?.replace(",", "")
val result = YahooRateData(
code,
price?.toDouble(),
changePrice?.toDouble(),
changePercent?.toFloat(),
bid?.toDouble(),
ask?.toDouble(),
lowPrice?.toDouble(),
highPrice?.toDouble(),
low52wPrice?.toDouble(),
high52wPrice?.toDouble()
)
return result
}
fun saveMainData(result: YahooUsdkrwAndMainData) {
if (result.isSuccess) {
// 기준이 되는 rate data 저장
result.usdkrw?.let { yahooRateRepository.save(it) }
if (result.mainData != null) {
// main data 저장
for (mainData in result.mainData) {
yahooMainRepository.save(mainData)
}
}
}
}
}
// vim YahooRateScheduler.kt
package com.dev.kopring00.external.fiat.schedulers
import com.dev.kopring00.external.fiat.services.YahooRateService
import org.slf4j.Logger
import org.slf4j.LoggerFactory
import org.springframework.beans.factory.annotation.Autowired
import org.springframework.beans.factory.annotation.Value
import org.springframework.scheduling.annotation.Scheduled
import org.springframework.stereotype.Component
import kotlin.random.Random
@Component
class YahooRateScheduler @Autowired constructor(
private val yahooRateService: YahooRateService
) {
// example : https://finance.yahoo.com/quote/KRW=X?p=KRW=X&.tsrc=fin-srch
@Value("\${api.url.yahooFinanceRate.usd1}")
private lateinit var apiUrl_1: String
@Value("\${api.url.yahooFinanceRate.usd2}")
private lateinit var apiUrl_2: String
@Value("\${api.url.yahooFinanceRate.usd3}")
private lateinit var apiUrl_3: String
val logger: Logger = LoggerFactory.getLogger(YahooRateScheduler::class.java)
// 미국 한국 환율과 일자별 주요 지수 정보 수집해 저장
fun updateMainDataWithUsdkrw() {
val apiUrl = apiUrl_1 + "KRW" + apiUrl_2 + "KRW" + apiUrl_3
val mainAndRateData = yahooRateService.getMainDataWithUsdkrwFromWeb(apiUrl)
if (mainAndRateData.isSuccess) {
yahooRateService.saveMainData(mainAndRateData)
} else {
logger.error("[ ExternalApi:YahooMain ] Fail save Data.")
}
}
// 매분 00초마다 크롤링 진행
@Scheduled(cron = "0 * * * * *")
fun scheduledUpdate() {
updateMainDataWithUsdkrw()
}
}
7. 데이터 수집 결과 예제
