Home	Login - Register		Downloads	Developers, news & Help	Donate
Lazy . Admins . Do . It . Yourself		Project areas:     Routines    Cluster networking    Control Panel    Web Site related					Work Log
Style: Light / Dark		Server time: Sep 04 2010  20:27:52

LADIY 2 full work development log

This log is a personal record of work done since starting development on LADIY version 2. The log works backwards with newest work displayed at the top of page down to oldest (first) log entries. Many times I would be writing entries after a long coding session and mostly around 6 A.M in the morning, so please excuse any bad spelling that may have been missed.

Key: CP = Control Panel 25th November Majority of LADIY site complete, all discussion area`s finished ready for use. "LADIY.php" added option to delete an ignored network stored machine. Option only visible when set too ignore and deleting requires confirmation stage yes no selection to ensure no accidental deleting occurs. Ran LADIY since 22nd November until now as final test run: *) Inserted bad IP`s into `l_cluster` table directly from SQL. LADIY shared the IP but all systems resulted in declaring the IP bad and set too ignore the host. *) Attempted to make successful contact from a none LADIY machine emulating POST, Even knowing protocol LADIY refused the connection due to bad handshake protocols. *) LADIY has been running for this time along side my daily activities generating high CPU loads, LADIY adjusted run time allowed processes accordingly each time. 15th October Bug with NetKeys being used including bad characters for HTTP send < > ! & / { } causing bad network passwords being decrypted and failing due to no access rights. Fixed all Higher ASCII to be replaced with exceptions and replaced back upon decryption. Whole system works flawlessly now. Since encryption some cases result in bad data sets for IP addresses so added IP check address is valid before further use via Regex preg_match and converting address to long ip2long and allow only if result. Network testing in progress again having issues with either network reliability or some other issue It`s rare but needs resolved. 13th October "Keygen.php" now includes all keys from "NetKeys.php" so when generating keys it`s using actual Ascii characters in use and also prevents having to update several files. Key set reduced to 85 keys. Discovered certain higher Ascii keys like & < > needed converted for data sending to prevent errors. 10th - 12th October Muddle() moved to "Lang_Engine.php" as complete. Created Encryption/Decryption engine Muddle() which acts as both: *) Engine uses 26 pre-set keys in file "NetKeys.php". *) Each key is an array represented by a letter in the alphabet a-z / 1-26. *) Each Key has all acceptable ascii characters that can be encrypted (92 in total). *) Engine randomly picks a key to use for EVERY data send (like rolling codes). *) Engine itself is useless without key sets. User can create own key sets using provided "keygen.php" file. *) Result is user unique keys (26) * 92 characters/key = 2392 usable characters for encoding with. *) "NetKeys.php" must be the same on each network LADIY machine however file can be copy pasted. "Keygen.php" outputs 26 randomly created keys in Array() format ready to copy paste directly to "NetKeys.php" 7th October ************** 5th and 6th October Ran LADIY full time on both laptop and PC inserting bogus network addresses which would be activated and ran for attempted contact Network Address Table updating between all live boxes and general build up on all statistic numbers to ensure calculations in scripts didn`t bug out on larger number sets. Both systems ran without errors or bugs resulting in the system being ready for encryption and decryption coding to be applied to all incoming and outgoing network traffic. 4th October Totally redesigned top menu of LADIY Grouped all routine option pages into a single drop down menu to select from Click to view that area. Result is able to select a menu width as little as 700 pixels wide excellent for smaller screens. All sub-area`s of the control panel altered to suit the 100% dynamic layout to allow the Master width to fully control size. 3rd October "Net_Cluster.php": *) Fixed bug for those machines being allowed to continue regardless of errors will not constantly log error reports if 1 has already been submitted. *) Added master override for any host as `userignore` User can set this option in Cluster Home. Net_Cluster will then totally ignore any host with `userignore` set to true. More importantly LADIY will resume contacting the box when the user sets `userignore` back to false (no). *) Routine and "Comm.php" communication file both take "Network enabled/disabled" into account prior to any run Routine will simply not run Any outside communication will yield no fail errors or anything when disabled. *) now able to handle bad connections by setting classes to the status If after 2 attempts there is no contact made its marked as having an error. After 10 errors the host is set to ignore so it`s no longer cycled however LADIY will take hosts with errors as those never checked 1st those checked but less times than others then move onto those nearing 10 errors last this way the system will not attempt the same machine for connection constantly in case there is some network connection error or such. The time between retries is set in `Lset.php`. 1st October Cluster home page now shows if any errors amount of errors and last error timestamp as set by LADIY. "Lang_Engine.php" edited to include GetNewHosts() as finished routine. *) Data2Host() edited to check fsockopen() status as statement of running to stop text error output from bad IP addresses. Created new Routine file "Net_Cluster.php" to handle all network updates invitations Network Address Table sharing and port scanning. *) System works in 3 stages of checking brand new IP`s added 1st those not new but have less than 3 errors 2nd and lastly everything else that has no errors. *) During each stage it will attempt to contact the machine for a invitation request and if successful continues to GetNewHosts() sharing the IP list with each other updating accordingly. *) "Net_Cluster.php" now fully working on cron updating all network machines connected with any and all information. *) A new host being contacted has 3 chances to respond after which the host is given up on. *) Whole routine is exempt from status changes as network times will always vary meaning creating an average time is pointless. "Status_Scan.php" added to ignore "Net_Cluster.php" although user can decide to use by commenting out a single line of code as marked in the code. 30th September Invite update section in Cluster Networking now uses GetNewHosts() at the same time of updating resulting in a brand new machine connecting to the network will be logged and receive all other known network machine addresses. Work almost finished on new function GetNewHosts() which sends LADIY`s own IP network list to the target host which then cross checks the given list against its own. The target will add any unknown IP`s to it`s own system and then sends back any IP`s the target has which the original sender does not. The original LADIY system will then add the new and close the connection. 20th September Fixed bug in Invite Control panel system can no longer attempt to invite the system to itself by using that boxes own IP. Applied too all steps of data acceptance up to point of send. 19th September New function UpdateHost() created using Invite system as base template to create the function. This now replaces all code in control panel operations for use and now LADIY can use it. Function holds switch so control panel can request text feedback while LADIY can request a simple True or False result. Passwords now used in every aspect of data transferring with exception of initial invitations to LADIY or "What`s my IP" request anything else will fail connection. Began work on new function GetHosts() which currently sends the systems all known network IP`s with exception of it`s own and that it`s "talking" too the remote host then checks the IP`s against it`s own. The target then adds any new unknown IP`s to itself and sends any new IP`s found back to the original host. 16th-18th September `Lset.php` created as master settings file included in all control panel and Cron Runtime.php Contains as many setting values that are repeated so there is only 1 point of editing. Cluster networking area: *) Network discover area added "Delete all unsaved scans" button to save time when lots of previous scans are listed Save the few wanted and batch delete the rest. *) Send Invite changed too Send Invite / Update info As now the same method can be used to manually update network information too any other LADIY machine. *) Invite/Update added select option to choose an already stored LADIY system rather than typing out IP`s if the intention is too update information. *) Had issue of PHP only detecting local IP address as localhost/127.0.0.1 rather than network broadcast address on local networks All methods tested don`t work on all operating systems Created function GetMyIP() which LADIY and control panel can use to ask any other LADIY system what its IP address is without any handshake so new systems not yet authenticated can still resolve their own IP. *) Network Discovery scan while scanning IF it detects port 80/web as open it will ask if it`s a LADIY system using new function RuLADIY() to ask. *) Invite/Update fully operational able to connect to any other LADIY system on it`s network and exchange network information about each box. *) Handshake system complete with originating host sending it`s IP as handshake in ip2long() format the Recipient retrieves the sent data and checks the originating IP of the sender to compare values with Hense IP detection is essential else handshake WILL fail. *) Comm.php is the network interface page used to receive data from other LADIY systems After handshake Comm.php now checks to see if the host requesting is in the cluster network table and unless its an invitation or IP request it must be logged otherwise the request is denied. 15th September Started work on network communications Initial contacts made Handshake developed for authenticity check on connecting machine. 14th September Adjusted routines so that when a routine is first detected as default the Objective and if a system routine with file info would be taken Now you can specify the timespan group to go in and Maximum status allowed in the routine file also. Result is auto set-up for Control panel and only leaving setting the routine to enabled left to become fully live and operational. System as it stands installed on both laptop + PC Both see each other in Network discovery. Newest PHP version causes any open close PHP tags <? ?> to have php stated each time So whole code adjusted. 7th September Whole system now running on its own custom SQL connection HTML header info and footer for DB close. 5th September Redesigned whole menu system of Control Panel to a single bar menu with most graphical links in place of full text links Each image has title text for mouse hover over. 4th September "Network Discovery": *) Added ability to scan a select range or group of IP`s *) Scan a single IP 3rd September All log displays with exception of default home log page (limited to user specified or default 14) now show options to navigate between pages forward/back and specify how many log entries to show per page. 29th August Routines: "SQL_Scan" "HDD_Scan" "LADIY_Scan" "Routines_Scan" "Status_Scan" and "Secure_Scan" all run silent unless there is an issue to report Otherwise log becomes flooded from routines running multiple times a minute. This option is still 2nd to the master log switch option and can be overridden. 28th August "Network Discovery" if `l_discover` table is missing (first time use etc) it will attempt to create the SQL table for the user if successful it will automatically refresh to be used. (PHP SQL privileges must include "create") 27th August "Network Discovery": *) Added "Scans" folder to "sub" folder in "routines" folder. *) Option to view image map of network scan link added to scans info selection areas. *) Scan map totally automatic layout depending on users width height hub size required. *) Scan map generates image to file in "Scans" folder with text file of Imagemap for the network scan. *) Imagemaps automatically generated on the fly for use. *) Viewing maps switched to 2 stage operation Check if map exists else create one. *) Options to generate new map layout (regardless of existing). *) Options to view map hubs with link lines between home and hubs as ON / OFF. *) Total process saves image re-generation for later viewing. *) Scans folder automatically created with .htaccess and index.html files if they don`t exist. 23rd August CP now includes "Lang_Engine.php" as default which includes "xss.php". Every single POST and GET data pass within CP is XSS checked. 22nd August "Network Discovery": *) Added option to save a network scan with `save` added to `l_discover` table *) Added option to delete a scan with confirm yes prior. *) Added option to select a single scan as the master scan template for LADIY to use in operations. *) Cannot delete a master scan or if last scan remaining. *) Notice shown on top of CP if no master template has been selected. *) Text output of scan now given also when viewing a previous scan. 19th August "Network Discovery": *) Requires button confirmation of scan wanted. *) Saves each scan in DB `l_discover` for previous report checking. *) Ability to read reports given on home of discovery page. *) All functions and CP for "Network Discovery" all within 1 file. 18th August "Network Discovery" added to Cluster CP area Will detect all known machines within its own IP range Try and determine if the machine is on-line if port80 is active > If so try LADIY communication Try TTL to ensure box is active within time limits IF nothing then try again on possible active services the machine could provide such as mail FTP and other server types. TTL() function updated ping execute line to include a time out period adjustable by the user otherwise it will hang the script until the system times it out. 17th August "Send Invite" developed further requests IP as step 1 checks IP exists and ports are open for use as step 2 Finally step 3 checks Time To Live average to machine and is ready to make contact. 16th August Cluster CP: *) First time or editing network settings: added password options to password cluster networks. *) New area "Send Invite" to send an invite to another LADIY machine to be detected on the network. `TTL` added to `l_cluster` table for working out shortest route. 15th August Created new `TTL` function which auto detects OS and pings a given IP 2 times takes the returned results and keeps the last line which states the average TTL in MS Strips everything and keeps just the average number to be used. 12th August Port check any port Function CheckPort($port) added to `Lang_Engine.php` Created new CP area linked by "Cluster" > LADIY Cluster networking: *) Created new table `l_cluster` *) If first time use of area Checks port80(web) is available if not cannot continue until resolved. *) First time use requires settings be selected before allowing any other options. *) Ability to re-edit settings. *) Cluster area home listing all stored LADIY machines in `l_cluster` Created Mini icons for use in the menu of control panel for each routine group. 10th August Ability to view SQL error report from View area of Control panel. Settings area of control panel renamed to `View` seeing the multi-view options now. `Secure_scan.php` now covers cron routines and sub-routines folders with `.htaccess` and `index.html` `xss.php` created inside folder `sub`: *) Added flag will only function if WantingXSS = 1 otherwise default 404 File has no output so would otherwise should blank white page which is not bad but indicates someone guessed correctly = bad. *) Function only included if not exists. *) RemoveXSS() will strip any injection technique/bad code from any POST or GET variables passed too it. *) Included from `Lang_Engine.php` so always available. Folder `sub` created within `routines` folder: *) Folder for use of sub-routines which master routines can call use include etc. SQL_Error_log function formatted to give break dashes between entries as ----- for clear view. 8th August 2009 `Lang_Engine.php` new Function SQL_Error_Log to report all SQL errors outside of web/htdocs directory to custom text file with 0 GUI output for intrusion attempts. Records date time stamp file line number error itself. `Secure_scan.php` created checks .htaccess and index.html files exist if not it will create them. This will mean any one of the following will all result in the exact same layout/code format "Roll another.." message: *) 404 incorrect page request *) Logged in > incorrect access rights *) Visitor guessing routine file names and directories. *) Even hitting the correct folder/file with any of the above missing. .htaccess file created in routines folder which redirects 404 (incorrect page) errors to index.html in the routines file directory which outputs "Roll another blunt...Nowt here". Each routine needs `Runtime.php` to work if someone tries to access a routine file by itself it will know `Runtime.php` is not present and give the same "Roll another blunt...Nowt here" like a 404 so it looks like an incorrect page request. 7th August 2009 Control panel: *) Code read from files is now colour coded for commenting to easily read over like in editors. *) New feature Settings which lists all routines physically present on the left when one is selected the settings from the selected file are shown in page by using the new settings flags. Changed format of settings being indicated in a routine file using #<SETTINGS> #</SETTINGS> 6th August 2009 Control panel given option to view new warnings if any When report viewed marked as read no longer new. Regular log method changed on all routines LADIY_Log() created in `Lang_Engine.php` which auto chooses log type based on settings in `Runtime.php` so routines logging is 1 line of code to operate. `SQl_Scan.php` tested again on several table errors found to send email per error Now collects as batch and if error sends single batch report. Report_Serious function given: *) New method of sending error message via syslog using SubMethod to specify LOG_alert type. *) Additional flag to specify a SubMethod to use. `Runtime.php`: *) Cleaned up as finished functions move to `Lang_Engine.php`. *) Reports log type added too settings Allows user to choose where regular logs go: Control panel log syslog or both by choose 12 or 3 as the setting. 5th August 2009 `Runtime.php` Server load function now supports Windows and Unix/etc with auto OS detect. `Lang_Engine.php`: *) Now fully accessible for use by any routine at runtime. *) Now included as critical routine file if missing cron will not run as critical error. *) created as base plate for time being with hold all critical communications routines. Control panel: *) will not show options at all for `Lang_Engine.php` routine Output only. *)Control panel for routines now indicates which are you are in prior to data on every option page. `HDD_Scan.php`: *) Given a minimum space limit which defines in KB how much minimum free space should be upheld. *) Able to trigger HDD_Cleaning function at runtime if minimum space breeched Intentionally blank to suit users needs of cleaning operation required to run. `SQL_Scan.php` can now: *) Now automatically grabs the database name to use SHOW TABLES opposed to SHOW TABLE STATUS. *) On disk full error(1021) set `HDD_Scan.php` to on next runtime to clear space. *) On Windows OS stop and start SQL if ran as a service on machine in error(1037) out of memory. 4th August 2009 `SQL_Scan.php` Uses Report_Serious() on unfixable or failed-fix table errors. `Runtime.php` now holds Report_Serious() function where a report and method of sending a error report can be selected and ran At the moment email is sent as method 1 but open to more as needed. `SQL_Scan.php` Error cases able to try Quick fixing SQL tables If fail able to try full Repair operation or if Error case seriously tries full repair first. Benchmarked `SQL_Scan.php` with 30MB of overhead which was detected and cleaned in 0.044 seconds. `Runtime.php` given new setting Master Log Switch which will over-ride every routines own logs enable/disable option. Meaning if the master log switch is set to disable then nothing at all will log in a sort of Stealth mode or simply just don`t want logging. ALL routines given a new setting to enable or disable if that routine logs or not Allowing for routines to just run with no output at all. `HDD_Scan.php`: *) Improved Unix support by scrapping the need for any shell file at all Instead running shell command to output the results to text for next runtime reading. (Thanks to irc.undernet.* #UnixTeam SuSEz who suggested new method) *) Added MB conversations to use in reports as well as KB totals for easier reference when looking. 3rd August 2009 Formatted all current files to a certain commenting code set as follows: *) All debug statments commented out when not in use are always at beginning of a new line. *) All comments use # not // *) Only comments which indicate an end of a script area like } use // like }//END if(x=y) *) All #comments are placed 1 space from end of commented string unless on a {} `HDD_Scan.php` now fully recording Windows based free drive space (on any specified drive letter) Unix based shell script needed but `HDD_Scan.php` does detect if Unix and will look for report file when in place. 2nd August 2009 `Runtime.php` cron error reports more in-depth for routines that where denied run access. `LADIY_Scan.php` altered so any routines not exceeding 0.5 seconds do not receive drastic status changes due to the extremely low runtime This was effected during scripts being developed and 0.005 seconds runtime being more than the previous 0.002 seconds was classed as a massive change. LADIY control panel given new menu area "Reset runtimes" allows any routines last runtime to be reset so it`s classed as expired and will be ran on next cron run. `HDD_Scan.php` created for Cleaning group Routine is able to run on Windows BSD Unix mac and Solaris due to auto detection of operating system defined by settings in routine. 1st August 2009 `SQL_Scan.php` now able to check tables by requesting no data but if failed retrieves mysql_errno and using the number selects what function it will perform. `TEST_script.php` created to test reporting HDD or folder space if required Currently reports for Windows and Unix using a custom shell script required to save a Unix commands output as single number as space for the script to read from a file within the Unix environment. 31st July 2009 `Status_Scan.php` changed so scripts not exceeding 0.5 seconds runtime are not given high status values. `SQL_Scan.php` checks database overheads and if exceeding given amount cleans overhead. `SQL_Scan.php` created for Cleaning group. 30th July 2009 `Runtime.php` now able to generate its own error reports if required to enter into the log. In page if<x>rame placed to the side of the log for reports to open in for easy oversight. LADIY 2 Control Panel Log display created; Colour coded entries report link and warning display. If a routine at runtime encounters errors it can generate a report and attach it too the log entry. All routines given ability to log at runtime for user GUI. `Runtime.php` changed so routines to be ran are calculated and added to array then called in a shorter loop after results in much quicker run times. 29th July 2009 Created new logging() function multifunctional for any aspect of regular site or LADIY routines logging. 28th July 2009 `LADIY_Scan.php` can now on errors for performance alter 2 critical cleaning routines at least 4 cron times old to be ran next in order to try clear issues. `Runtime.php` given now able to choose to run a disabled high status script IF its within last 2 routines to be ran at cron time and there is double the amount of time required left for the routine as a time buffer before cron resets. `LADIY_scan.php` created to check LADIYs own critical routine files and preform tasks: *) Check routines status for extremely high status records and if too high apply a specified delay in seconds. *) Ability to set a warning to routines so if 2 cron runs results in warnings then routine is performing badly otherwise warning removed on next run with 0 errors. Added function to `Runtime.php` that if `LADIY_scan.php` `Routines_scan.php` or `Status_scan.php` are missing it will refuse to run as critical files are missing. `delay` values added to `routines` table so LADIY may apply a delay on a routines runtime. `CPU` value added to `routines` table so required routines may benchmark CPU value at time of run 27th July 2009 `Runtime.php` fully operational running routines prioritises order of run counting runtime checking runtime status and able dismiss running routines based on data at runtime. `Status_Scan.php` average benchmark run time = 0.002 seconds. `Runtime.php` now counts routines benchmark times to calculate total cron runtime: *) If a Routines status is slightly above maximum allowed then `Runtime.php` can see if the routine is in the last 2 to be ran in that cron run IF the routine is within the allotted last 2 AND `Runtime.php` calculates enough cron time left + A margin of error time buffer just in case it takes longer again THEN allow the routine to run to acquire new benchmark time and update status. All status reporting functional and being present within LADIY 2 control panel. Essential control of routines via control panel all functional. Routines_Scan.php and Status_Scan.php as 2 critical functions fully operational. Created new cron page for server to activate: Below examples represent only that file running 0 other operations OR same as nothing due to run. *) NEW `Runtime.php` cron from LADIY V2 = 34 lines of code to run per cron run time *) Last "intel.php" cron from LADIY V1 due to design was over 900 lines of code to run per cron run time 26th July 2009 LADIY control panel now able to show status tracking as +/- changes between running to show average direction of run times status. `routines` SQL table modified with with extra status benchmark row `Status_Scan` beta now benchmarks each routine as % by: *) Last benchmark run time. *) Total runtime / total times ran. *) Average comparison between Last run time and ( last runtime + 2nd last runtime / 2) *) MIN / MAX comparisons between above benchmarks against Last run time. 2nd Routine created `Status_Scan.php` to scan all Enabled and Physically present routines. Status.txt as draft ideas + maths algorithms for calculating status. Routine Group `Brains` renamed to Group `Critical`. 25th July 2009 `routines` SQL table modified with several more rows for benchmarking and status tracking. Primary Routine groups given options on each routine row to set timespan between routine being ran status allowed (1-10) before routine considered running critical enabled/disabled and the Group of which the routine belongs too all in 1 form action for applying settings and new routines quickly. 24th July 2009 All routines dropped into "/routines/" folder are now automatically set as Disabled with Control panel page options to view enabled and disabled as groups + options to set opposite effect quickly. (those in enabled can be quickly disabled and vice versa) 2 day test run on basic routine operations completed Randomly dropping and removing routines from "/routines/" to ensure stability and low benchmark times - 0.0015 to 0.0039 seconds average on 100 routines in folder being scanned. 21st July 2009 Confirm patch on deleting routines entries has manual 2nd choose yes AND routines must NOT be physically in routines folder or deny access. Ignored routines listed in own group area where options to UN-mark as ignored OR Delete the routine entry. Routines no longer detected as being present/missing are automatically moved to own missing routines page where routines can be set too ignored if a routine was taken out to be worked / reason. Routines based on physical files being present to run so "Routines_Scan.php" default marks all routines are NOT being present prior to running and locks during runtime to stop accidental access while set to not being physically present even though it maybe. 20th July 2009 Routines report all benchmark results from within each routine leaving runtime shorter if not ran. Able to view routines in Individual groups for easier oversight of a group. All routines now able to be added to 1 of 6 master groupings for routines. Groups ATM: Brains A.I Security Networking Cleaning and Site based. Routines set a standard so "Routines_Scan.php" can requests routines information. Routines_Scan.php fully indexing "/Routines/" folder based on File last modified tags. First Routine created and manually ran "Routines_Scan.php" 1st concept of `routines` table created. LADIY 2 Control Panel created showing "/Routines/" folder as list in a single page. Routines folder created in LADIY area. 18th July 2009 Created LADIY 2 Language function as document text for beta reference later. LADIY Version 2 language set 1.0.doc 16th July 2009 Started work on DFD for LADIY but got bored/ headache and got high instead. Wrote first documents planning basic structure system of new LADIY 2. LADIY Version 2.doc

Open Source

www.LADIY.info     About