Did some minor scripting today, forgot how bad I was at scripting.  Trying to parse this JSON formatted text was like a nightmare.  Trivial for someone
who does this all the time, I am sure.  I learned and re-learned a few things I had forgotten.  Some had nothing to do with the task (parse the alienvault OTX security feed, which is in Json or some wierd text format).

I learned:
XmlStarlet Command Line XML.   Wow that looks like a super handy way to parse an XML formatted file.  Too bad the file I was working on really wasn't XML.  Opps.

How to set a unix file to execute mode.  DUh, I knew this like a few years ago, but forgot you need to set UNIX files with the EXECUTE bit, if you want to run scripts.  Unix security,  do a "chmod 777"  to set the file "" to execute.

For text processing, I can do a few things in DOS with the "for" command, and process chunks of data.  Done it for years, but some things you need super text edit power of unix.  I used SED !  Sed (stream editor) is a awesome tool for text processing.  After much reading about the different things sed can do, I found the right switch.  Here I had a huge single line file.  I need to break it apart into lines for further processing.  I choose the field, and told SED to put a line break.
To add newline AFTER every word that has "IPv4" in the file otx.txt, and create a new file (line.txt), simple enough that I used this:
root@venture:~# sed 's/IPv4/&\n/g' <otx.txt >line.txt

Some people can hack json with Java or Python, god bless em.  I can't, so I use a bunch of different commands that I learned long ago.  Here is how I manipulate alienvaults OTX feed.  This is a unix script, by the way.

to run, you might have to type "./"

root@venture:~# cat
rm o*.txt
curl -k -H "X-OTX-API-KEY: 7e43a43a2ca1f80ef08db491e899c1a966eefe20df0" > otx.txt
sed 's/indicator/&\n/g' <otx.txt >oline.txt
grep IPv4 oline.txt > otxipv4.txt
gawk -F'"' '{ print $3 }' otxipv4.txt > oipv4.txt
cat oipv4.txt >> ipv4list.txt

This script takes alienvaults complicated, very heavy and detailed output, and simplifies.  For me, I only wanted the "bad actor ipv4 addresses".  I will take those ip addresses, put them on a page here and be able to pull them into my various network equipment.  otxipv4.txt has just the IPv4 related events, and the date.  I will change the script to store these files, so I have a record of how far the events go back (how many I missed).  Eventually hope to programmatically look at the date string, and only get the events since a certain date. 

So my distallation of alienvaults feed, keep in mind it's custom for me and only has the default alienvault data and a few other subscribers.  You should make your own custom feed by using alienvault, and choosing security information relative to you. 
If you want to see my current IPv4 list, click here.

The final output, ipv4list.txt, is only a list of IP addresses, separated by line feeds.  Just what paloalto firewall wants, read here about dynamic block lists

Using Expect
Modify cisco/linux/sonicwall/ any type of CLI using Expect. So there is a several hundred page book you can buy about 'expect'. I bought a used copy for 4 dollars, but like most IT people, time is of the essence. So I tried to get expect working by myself and then later with the help of a co-worker. Here is what I learned. My main stumbling block was trying to get arguments into EXPECT. So I had no problem making a expect script. I started off by looking at examples on the web and also the script that AUTOEXPECT creates when logging into a system. Once I had that basic knowledge, I was able to pass some variables to expect, and those to the remote system (in this case, a sonicwall firewall). So here we go. In the below example, you see, then three (3) strings. Those strings are passed into expect using the lindex command and 'set xxxx' command. So the only way I could get a variable from bash, so far, was to use this method.
. Example:

-bash-4.1$ ./ Jan2**** HIE_SUPER
Notice the “set commands” below. I forgot about them, I read about them last week, dang it, that’s how you do it.

#!/usr/bin/expect -f
set pass [lindex $argv 0];
set desc [lindex $argv 1];
set ip [lindex $argv 2];
spawn ssh dr2@
expect "Password:"
send "$pass\r"
send "\r"
expect "dr2@CLDLV-IBF-CAGE5017-01"
send "config\r"
expect "config(CLDLV-IBF-CAGE5017-01)# "
send "address-object ipv4 \"$desc\" host $ip zone WAN\r"
expect "config(CLDLV-IBF-CAGE5017-01)# "

Here is some other examples of expect (on stackoverflow). I could not get one of these to work (maybe because of differences in BASH / operating system ? SSH sessions using expect
Your Website
Your Company Slogan here ...
WYSIWYG Web Builder

Latest News