Streaming telnet logger to MySQL database, Linux

[gregulator]

I have some temp sensors in the building that have either a web interface, or stream output over a telnet session. Initially I looked into screen scraping the web interface with BeatifulSoup, but it updates with javascript functions so I can't scrape the temp numbers (looks like I could use selenium, but I moved on).

It also streams a continuous output to a telnet session, and I can pipe it to a "tee" and output that text to file. This is my current approach. If I want to log the data to a MySQL database, is my best bet to:

1) Write a script to run the telnet session, logging the output to file
2) Create another script/daemon to consume the text file and put it into the MySQL database?

I am not a Linux user, so I don't know if this is the best way. Thanks!

 

[sathyan]

I would probably use Python for this with the modules telnetlib and mysql.connector. There's a function conn.read_until(string) that can watch for strings and then respond with conn.write - you use this to login (but could also watch for a termination in a loop if you will need to do multiple separate inserts)
http://docs.python.org/3.3/library/telnetlib.html
then do str = conn.read_all()
parse str as needed and load to your database

 

[gregulator]

Thanks for the suggestion. I tried that library, and it works okay, but I am having a problem with getting the text I need. Unfortunately, the text stream does not concatenate, it actually erases the existing text and overwrites itself and it seems like it is hard to grab with the telnetlib functions (I tried a few different ways, but I couldn't seem to get it to work.. Linux NOOOOOOB). It is easier for me to grab with tee so I think I will go this route for now. Your advice is much appreciated though!!!

 

[Crusty]

Any scripting language that has a mysql library will work.

Just pipe the output of tee to the script you write, and have the script read stdin for the text to parse and then stuff it into a database.

 

[JasonCoder]

If the web interface updates with Javascript routines, perhaps look at what the javascript data sources are and query those. Might be a simple JSON service you could hook up to.

 

[gregulator]

Took me forever to get it to just write to a file in real time, this was such a pain:

telnet 192.168.1.100 2000 | stdbuf -o0 tr '\b\r' '\n' | grep -o --line-buffered '[T,H,D]....[%,F]' > test.txt

Oh the joys of relearning Linux every time I use it! Now on to the consuming side and putting it into a DB...