NewlogFetcher

java.lang.Object
- org.qbroker.common.NewlogFetcher

```
public class NewlogFetcher
extends java.lang.Object
```
NewlogFetcher reads the new log entries from a sequential log file

Use NewlogFetcher to check a sequential log file repetitively. It only picks up the newly updated log entries. It guarantees the order and the integrity of the new logs. In order to guarantee the order and the integrity, each log entry should have a timestamp and the value of the timestamp has to be in non-decreasing order, i.e., the timestamp of the log is always no later than those of the following logs.

In order to keep tracking the positions, NewlogFetcher relies on a reference file. The reference file contains the position of the reference log, its timestamp and the offset of the objective log. A reference log is a log entry with a valid timestamp and is used as the reference to locate the objective log. It may or may not be in the current log file. The objective log is the first log entry in the current log file that has the timestamp later than the reference timestamp. The offset is the byte count from the beginning of the objective log to end of the last processed log entry. Any log after the offset is treated as a new log. In fact, NewlogFetcher frequently opens the log file and looks for the objective log. It then scans the offset bytes to locate the new logs.

Here are the rules for the reference file:

(1) If there is the reference file, use it and trust it.
(2) If there is no reference file, use the timestamp of the current log file as the timestamp of the reference log and set the position of reference log and the offset to 0.
(3) If there is no current log file, try the timestamp of the old log file.
(4) If still no reference, use the default reference (0 currentTime 0).
(5) If there is a mismatch to the position of the reference log, trust the timestamp of the reference log and search for the objective log in the current log file. The objective log is the first log entry with a timestamp later than the reference timestamp.
(6) If there is a mismatch to the offset of the objective log, trust the current log first.
(7) If it does not find a objective log, reset the offset to 0.

The reference file is not required. But if the file is not defined, you will have to set SaveReference to false to disable the update on the state info.

To speed up sequential IO, it has opened two IO Streams on the same FD. RandomAccessFile is used for random IO and seek. BufferedReader is used for sequential IO. The speed-up is more than 10 times.

Author:

yannanlu@yahoo.com

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`DEBUG_BISECT`
`static int`	`DEBUG_INIT`
`static int`	`DEBUG_LINE`
`static int`	`DEBUG_LOCATE`
`static int`	`DEBUG_NEWLOG`
`static int`	`DEBUG_SAVE`
`static int`	`DEBUG_SCAN`
`static int`	`DEBUG_TIME`
`java.util.List<java.lang.String>`	`logBuffer`
`static int`	`MAXLOGLENGTH`
`static int`	`MAXNUMBERLOGS`
`static long`	`YEARMILLIS`

Constructor Summary

Constructors
Constructor and Description

NewlogFetcher(java.util.Map props)

Constructors
Constructor and Description
`NewlogFetcher(java.util.Map props)`

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`void`	`close()`
`java.lang.String`	`getLine()` It returns a line as a String read from the input.
`java.util.List<java.lang.String>`	`getNewlogs()` It opens the log file and locates the first new entry according to the reference info, reads the all log entries from the location line by line up to the end of the logfile or until the buffer is full.
`long`	`getOffset()`
`java.lang.String`	`getPath()`
`long`	`getPosition()`
`long`	`getTimestamp()`
`long`	`getTimestamp(java.lang.String line)` returns the timestamp in msec for the log entry or -1L otherwise
`long`	`locateNewlog()` It locates the first new log entry according to the reference info of the logfile and returns the position of the EOF or new log entry.
`void`	`saveReference()` It flushes the current reference info to a given file for log tracking only if the SaveReference is enabled.
`void`	`saveReference(long position, long timestamp, long offset)` It flushes the specified reference info to the reference file for log tracking no matter the SaveReference is enabled or not.
`void`	`seek(long pos)`
`void`	`setReference(long position, long timestamp, long offset)` It sets the current reference info.
`void`	`updateReference(long offset)`
`void`	`updateReference(long position, long timestamp, long offset)` It updates the current reference info.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - logBuffer
```
public java.util.List<java.lang.String> logBuffer
```
  - MAXNUMBERLOGS
```
public static final int MAXNUMBERLOGS
```
    See Also:
    
    Constant Field Values
  - MAXLOGLENGTH
```
public static final int MAXLOGLENGTH
```
    See Also:
    
    Constant Field Values
  - YEARMILLIS
```
public static final long YEARMILLIS
```
    See Also:
    
    Constant Field Values
  - DEBUG_INIT
```
public static final int DEBUG_INIT
```
    See Also:
    
    Constant Field Values
  - DEBUG_LOCATE
```
public static final int DEBUG_LOCATE
```
    See Also:
    
    Constant Field Values
  - DEBUG_BISECT
```
public static final int DEBUG_BISECT
```
    See Also:
    
    Constant Field Values
  - DEBUG_SCAN
```
public static final int DEBUG_SCAN
```
    See Also:
    
    Constant Field Values
  - DEBUG_SAVE
```
public static final int DEBUG_SAVE
```
    See Also:
    
    Constant Field Values
  - DEBUG_LINE
```
public static final int DEBUG_LINE
```
    See Also:
    
    Constant Field Values
  - DEBUG_TIME
```
public static final int DEBUG_TIME
```
    See Also:
    
    Constant Field Values
  - DEBUG_NEWLOG
```
public static final int DEBUG_NEWLOG
```
    See Also:
    
    Constant Field Values
- Constructor Detail
  - NewlogFetcher
```
public NewlogFetcher(java.util.Map props)
```
- Method Detail
  - locateNewlog
```
public long locateNewlog()
                  throws java.io.IOException
```
    It locates the first new log entry according to the reference info of the logfile and returns the position of the EOF or new log entry. Reference info contains file position and timestamp of the reference log, and the offset from the objective log.
    
    Throws:
    
    java.io.IOException
  - getNewlogs
```
public java.util.List<java.lang.String> getNewlogs()
                                            throws java.io.IOException
```
    It opens the log file and locates the first new entry according to the reference info, reads the all log entries from the location line by line up to the end of the logfile or until the buffer is full. It stores them in the logBuffer. It closes the log file and returns the List of new logs.
    
    Throws:
    
    java.io.IOException
  - setReference
```
public void setReference(long position,
                         long timestamp,
                         long offset)
```
    It sets the current reference info. It is required that you have to provide all three arguments
  - updateReference
```
public void updateReference(long position,
                            long timestamp,
                            long offset)
```
    It updates the current reference info. Use it only after you have found a new log. You have to provide all three arguments
    
    position: beginning of the new log entry timestamp: timestamp of the new log entry offset: length of the new log entry
  - updateReference
```
public void updateReference(long offset)
```
  - saveReference
```
public void saveReference()
                   throws java.io.IOException
```
    It flushes the current reference info to a given file for log tracking only if the SaveReference is enabled.
    
    Throws:
    
    java.io.IOException
  - saveReference
```
public void saveReference(long position,
                          long timestamp,
                          long offset)
                   throws java.io.IOException
```
    It flushes the specified reference info to the reference file for log tracking no matter the SaveReference is enabled or not. It is designed for public to control the log tracking only. So you have to provide all three arguments with care and disable the SaveReference to prevent the reference log from overwritten by NewlogFetcher itself. Please also make sure the reference file is defined.
    
    Throws:
    
    java.io.IOException
  - getTimestamp
```
public long getTimestamp(java.lang.String line)
```
    returns the timestamp in msec for the log entry or -1L otherwise
  - getTimestamp
```
public long getTimestamp()
```
  - getPosition
```
public long getPosition()
```
  - getOffset
```
public long getOffset()
```
  - getPath
```
public java.lang.String getPath()
```
  - getLine
```
public java.lang.String getLine()
                         throws java.io.IOException
```
    It returns a line as a String read from the input. The char of newline, '\n', will be at the end of the line. In case of empty input, it returns null.
    NB. make sure the buffer is reset at close of the InputStream It is not MT-Safe due to buffer, leftover, and startBytes
    
    Throws:
    
    java.io.IOException
  - seek
```
public void seek(long pos)
          throws java.io.IOException
```
    Throws:
    
    java.io.IOException
  - close
```
public void close()
```

Class NewlogFetcher

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

logBuffer

MAXNUMBERLOGS

MAXLOGLENGTH

YEARMILLIS

DEBUG_INIT

DEBUG_LOCATE

DEBUG_BISECT

DEBUG_SCAN

DEBUG_SAVE

DEBUG_LINE

DEBUG_TIME

DEBUG_NEWLOG

Constructor Detail

NewlogFetcher

Method Detail

locateNewlog

getNewlogs

setReference

updateReference

updateReference

saveReference

saveReference

getTimestamp

getTimestamp

getPosition

getOffset

getPath

getLine

seek

close