• Home
  • About

    • Kira's Blog Site

      Doctor X's blog site

    • Learn More
    • Email
    • Twitter
    • Github
    • Steam
    • Weibo
  • Posts
    • All Posts
    • All Tags
  • Data Analysis
  • Climate Projects
  • Videos
  • Life
  • Notes
  • Algorithm
  • Papers

Crawl in My MacOs

18 Feb 2020

XueQiu_Stock_Spider

  • 雪球网股票数据Scrapy爬虫
  • 生成数据
  • 分析数据
  • 预测股价.
# 这个爬虫是根据
$ git clone https://github.com/TrojanAZhen/XueQiu_Stock_Spider.git
# 进行修改的，修改过后的版本可以在
$ git clone https://github.com/Clynie/XueQiu_Stock_Spider.git

README:

雪球网 => redis => Scrapy爬虫 => MongoDB => 分析数据 => 模块 => 股价

设计流程:

  • 基于python 2.7, Redis和MongoDB, 使用Scrapy和Scrapy_redis以及mongo 模块.

  • 开启mongod和redis-server服务, 然后使用scrapy crawl雪球网爬虫

  • 打开scrapySpider/spider, python运行stock.py将stock_price的url列表发送给redis-server

  • Scrapy爬虫将从redis-server里面读取urls, 然后下载web数据, 再把数据存档mongodb.

  • 把股票爬取的基本信息, 再用python运行stock.py stock_price, 抓取股票价格信息.

  • python运行stock.py model, 分析数据, 培养模型, 预测股价.

建立python2.7 的环境变量


conda create --name complex_network Python=python2.7
conda activate complex_network
# add the env into jupyter
python3 -m ipykernel install --user --name=complex_network

The python2.7 required packages is

requests
sklearn
pymongo
redis==3.0
url==0.2

安装Redis：

type below:

$ brew update
$ brew install redis

To have launchd start redis now and restart at login:

$ brew services start redis

Or, if you don’t want/need a background service you can just run:

$ redis-server /usr/local/etc/redis.conf

Location of Redis configuration file.

$ /usr/local/etc/redis.conf

Get Redis package information.

$ brew info redis

Test if Redis server is running.

$ redis-cli ping

If it replies “PONG”, then it’s good to go!

Location of Redis configuration file.

$ /usr/local/etc/redis.conf

Uninstall Redis and its files.

$ brew uninstall redis
$ rm ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

Launch Redis on computer starts.

$ ln -sfv /usr/local/opt/redis/*.plist ~/Library/LaunchAgents

Start Redis server via “launchctl”.

$ launchctl load ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

Start Redis server using configuration file.

$ redis-server /usr/local/etc/redis.conf

Stop Redis on autostart on computer start.

$ launchctl unload ~/Library/LaunchAgents/homebrew.mxcl.redis.plist

MongoDB

Before you start to run the code, first of all, make sure the mongod is already installed and run.

If not installed, please try:

brew update
brew install mongodb
mkdir -p /data/db
sudo mongod

If the mongodb is already installed, just:

sudo mongod

下面是另一个爬虫：

StockSpider

git clone https://github.com/Clynie/StockSpider-1.git

README:

股票数据爬虫+分析+可视化框架

爬取的数据是这样的：

这是一个test

test_1


Like Tweet +1