Java: RegEx: WebSpider

In this tutorial we will create a basis for a WebSpider program that fetches all links from the given Web page, checks which have extensions of interest and downloads these files.

The result of the app will be a folder with Google Earth map overlays:


/Users/uki/Desktop/KMZ
├── 1.kmz
├── 10.kmz
├── 11.kmz
├── 12.kmz
├── 13.kmz
├── 14.kmz
├── 15.kmz
├── 16.kmz
├── 17.kmz

├── 18.kmz


We will use basic java networking classes:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;

and regular expressions:

ublic class RegExConstants {

   /**    * @param extensions    * @return    *  (                -- start of main grouping    *     [^\s]+        -- must contains one, or many strings (but not white space)    *     (             -- start of extension grouping    *        \.         -- existence of a dot, eg.: .kmz    *        (?i)       -- NOT case sensitive for the next group    *        (kmz|kml)  -- kmz OR kml strings    *     )$            -- should exist on the end    *  )                -- end of main grouping    */   public static String fileExtensionPattern(String[] extensions) {

      StringBuilder sb = new StringBuilder("");

      if (extensions.length > 0) {
         int n = 0;
         for (String extension : extensions) {
            if (n > 0) {
               sb.append("|"); // append OR            }
            sb.append(extension);
         }
      }
      String pattern = "([^\\s]+(\\.(?i)(" + sb.toString() + "))$)";
      System.out.println("fileExtensionPattern: " + pattern);
      return pattern;
   }

   public static final String anchorTagPattern = "<a *href=\"(.+?)</a>";
   public static final String urlPattern = "(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]";
}


As an Amazon Associate I earn from qualifying purchases.

Java: RegEx: WebSpider

In this tutorial we will create a basis for a WebSpider program that fetches all links from the given Web page, checks which have extensions of interest and downloads these files.

The result of the app will be a folder with Google Earth map overlays:


/Users/uki/Desktop/KMZ
├── 1.kmz
├── 10.kmz
├── 11.kmz
├── 12.kmz
├── 13.kmz
├── 14.kmz
├── 15.kmz
├── 16.kmz
├── 17.kmz

├── 18.kmz


We will use basic java networking classes:
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;

and regular expressions:

ublic class RegExConstants {

/** * @param extensions * @return * ( -- start of main grouping * [^\s]+ -- must contains one, or many strings (but not white space) * ( -- start of extension grouping * \. -- existence of a dot, eg.: .kmz * (?i) -- NOT case sensitive for the next group * (kmz|kml) -- kmz OR kml strings * )$ -- should exist on the end * ) -- end of main grouping */ public static String fileExtensionPattern(String[] extensions) {

StringBuilder sb = new StringBuilder("");

if (extensions.length > 0) {
int n = 0;
for (String extension : extensions) {
if (n > 0) {
sb.append("|"); // append OR }
sb.append(extension);
}
}
String pattern = "([^\\s]+(\\.(?i)(" + sb.toString() + "))$)";
System.out.println("fileExtensionPattern: " + pattern);
return pattern;
}

public static final String anchorTagPattern = "<a *href=\"(.+?)</a>";
public static final String urlPattern = "(https?|ftp|file)://[-a-zA-Z0-9+&@#/%?=~_|!:,.;]*[-a-zA-Z0-9+&@#/%=~_|]";
}


As an Amazon Associate I earn from qualifying purchases.

Project Planning

"Planning is a good preparation for an inevitable change of direction that throws out any plans made." ~Uki


As an Amazon Associate I earn from qualifying purchases.

Java: running with Maven



Compiling with Maven

note:

  • mvn clean install
  • location of created jar



uki@ WebSocketClientServer $ mvn clean install

[INFO] Scanning for projects...

[INFO]                                                                         

[INFO] ------------------------------------------------------------------------

[INFO] Building WebSocket-GlassFish 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

...

.to /Users/uki/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------







Using variables to store long values



uki@ ~ $ export WEBSOCKET_JAR=~/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jaruki@ ~ $ echo $WEBSOCKET_JAR/Users/uki/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jar




Starting SERVER



$ java -cp $WEBSOCKET_JAR edu.clcillinois.websocket.server.WebSocketServer -h localhost -p 8025




Setting homeName to: localhostStarting the server.Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.ServerContainerFactory createINFO: Provider class loaded: org.glassfish.tyrus.container.grizzly.GrizzlyEngineMar 03, 2015 3:56:54 PM org.glassfish.grizzly.http.server.NetworkListener startINFO: Started listener bound to [0.0.0.0:8025]Mar 03, 2015 3:56:54 PM org.glassfish.grizzly.http.server.HttpServer startINFO: [HttpServer] Started.Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.Server startINFO: WebSocket Registered apps: URLs all start with ws://localhost:8025Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.Server startINFO: WebSocket server started.
Type 'q' and Enter to stop server: 



Starting Client(s)




uki@ CNH_PROD $ java -cp $WEBSOCKET_JAR edu.clcillinois.websocket.client.WebSocketClient -h localhost -p 8025
[DEBUG] 2015-03-03 16:02:44,354 edu.clcillinois.websocket.client.WebSocketClient main - Starting WebSockets clientSetting homeName to: localhost



Monitoring Java processes and Killing them


$ ps | grep java
30227 ttys001    0:00.00 grep java

30203 ttys002    0:01.37 /usr/bin/java -cp ~/.m2/repository/edu/clcillinois/cit137/xyz.jar edu.clcillinois.websocket.server.WebSocketServer -h localhost -p 8025

$ kill -9 30203 uki@ SimpleWebSocketServer $ ps | grep java30230 ttys001    0:00.00 grep java


Maven pom.xml example

note: 
  • group id
  • artifact id
  • version
  • start-class















Maven Dependencies


note:
  • group id
  • artifact id
  • version








Maven build


note: 
  • plugins 
  • Java version 1.6, 1.7 or 1.8
  • TODO change to Java 1.6 and observer what fails
  • jar with dependencies









As an Amazon Associate I earn from qualifying purchases.

Java: running with Maven



Compiling with Maven

note:

  • mvn clean install
  • location of created jar



uki@ WebSocketClientServer $ mvn clean install

[INFO] Scanning for projects...

[INFO]                                                                         

[INFO] ------------------------------------------------------------------------

[INFO] Building WebSocket-GlassFish 1.0-SNAPSHOT

[INFO] ------------------------------------------------------------------------

...

.to /Users/uki/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jar

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------







Using variables to store long values



uki@ ~ $ export WEBSOCKET_JAR=~/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jaruki@ ~ $ echo $WEBSOCKET_JAR/Users/uki/.m2/repository/edu/clcillinois/cit137/WebSocket-GlassFish/1.0-SNAPSHOT/WebSocket-GlassFish-1.0-SNAPSHOT-jar-with-dependencies.jar




Starting SERVER



$ java -cp $WEBSOCKET_JAR edu.clcillinois.websocket.server.WebSocketServer -h localhost -p 8025




Setting homeName to: localhostStarting the server.Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.ServerContainerFactory createINFO: Provider class loaded: org.glassfish.tyrus.container.grizzly.GrizzlyEngineMar 03, 2015 3:56:54 PM org.glassfish.grizzly.http.server.NetworkListener startINFO: Started listener bound to [0.0.0.0:8025]Mar 03, 2015 3:56:54 PM org.glassfish.grizzly.http.server.HttpServer startINFO: [HttpServer] Started.Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.Server startINFO: WebSocket Registered apps: URLs all start with ws://localhost:8025Mar 03, 2015 3:56:54 PM org.glassfish.tyrus.server.Server startINFO: WebSocket server started.
Type 'q' and Enter to stop server: 



Starting Client(s)




uki@ CNH_PROD $ java -cp $WEBSOCKET_JAR edu.clcillinois.websocket.client.WebSocketClient -h localhost -p 8025
[DEBUG] 2015-03-03 16:02:44,354 edu.clcillinois.websocket.client.WebSocketClient main - Starting WebSockets clientSetting homeName to: localhost



Monitoring Java processes and Killing them


$ ps | grep java
30227 ttys001    0:00.00 grep java

30203 ttys002    0:01.37 /usr/bin/java -cp ~/.m2/repository/edu/clcillinois/cit137/xyz.jar edu.clcillinois.websocket.server.WebSocketServer -h localhost -p 8025

$ kill -9 30203 uki@ SimpleWebSocketServer $ ps | grep java30230 ttys001    0:00.00 grep java


Maven pom.xml example

note: 
  • group id
  • artifact id
  • version
  • start-class















Maven Dependencies


note:
  • group id
  • artifact id
  • version








Maven build


note: 
  • plugins 
  • Java version 1.6, 1.7 or 1.8
  • TODO change to Java 1.6 and observer what fails
  • jar with dependencies









As an Amazon Associate I earn from qualifying purchases.

apt quotation..