Commit Graph

158 Commits

Author SHA1 Message Date
Stefan Schantl
504fb53bcc ids-functions.pl: Add remove_from_etags() function.
This function is used to drop the stored etags data of a given provider.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-26 05:23:44 +02:00
Stefan Schantl
e41ee3e0f2 ids-functions.pl: Avoid suricata from loading rulesfiles of an
unsupported provider.

Modify the write_used_rulefiles_file() function to skip the rulesfiles
of unsupported providers.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-22 05:31:28 +02:00
Stefan Schantl
c62121c7e4 ids-functions.pl: Try to enumerate the dl_rulesfile if a provider is not
supported anymore.

In this case the details about the file suffix is not available in the
ruleset-sources file anymore. In this case now the function tries to
enumerate the correct filename.

This allows to display the correct stats in the WUI and to extract and
use the downloaded ruleset of the provider until it got deleted by the
user.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-20 20:58:04 +02:00
Stefan Schantl
38cf581405 ids-functions.pl: Remove temporary files if the downloader aborts.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-17 16:38:21 +02:00
Stefan Schantl
a2c56ead73 ids-functions.pl: Remove read_enabled_disabled_sids_file() function.
Not longer needed and therefore dead code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-15 06:02:49 +02:00
Stefan Schantl
a15c9b16b4 IDS: Move autoupdate logic to cron.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-15 05:59:33 +02:00
Stefan Schantl
b570d35c0a ids-functions.pl: Change location for downloaded rulesfiles to
"/var/cache/suricata/".

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-15 05:19:20 +02:00
Stefan Schantl
4f513522fe ids-functions.pl: Do not use a hard-code temporary download location.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-15 05:18:37 +02:00
Stefan Schantl
990d111d70 ids-functions.pl: Add support for Etags.
Etags are used to itentify if an ressource has been changed
by sending a special request and an Etag value to the server.

If the ressource has changed the server will serve the new content
otherwise it will return the 304 (Not-Modified) code.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-11 05:48:17 +02:00
Stefan Schantl
8d6714edc8 ids-functions.pl: Change backend to use one file to load the used
rulefiles.

Suricata seems to struggle when using multiple and/or nested includes in
the same config section. This results in a only partially loaded
confguration where not all rulefiles are loaded and used.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-04-09 14:10:13 +02:00
Stefan Schantl
849fc8ea15 ids-functions.pl: Drop oinkmaster related functions and declarations.
They are not longer needed and safely can be dropped.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:45:17 +01:00
Stefan Schantl
2deba6bf4a ids-functions.pl: Use "enabled/disabled" to mark if a rule should be
altered.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:36:24 +01:00
Stefan Schantl
794469483f ids-functions.pl: Replace call of external oinkmaster.pl to newly
introduced process_ruleset function.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:35:48 +01:00
Stefan Schantl
5a6c7bbe85 ids-functions.pl: Add process_ruleset() function.
This function is going to replace the part which currently the
oinkmaster.pl script does.

It will read in the extracted ruleset, remove duplicates and alter the
rules to alert or drop in case they match. Also rules will be enabled or
disabled if the used requested this.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:34:30 +01:00
Stefan Schantl
518cbdd389 ids-functions.pl: Add get_provider_ruleset_modifications_file().
This function will obosolete the old oinkmaster modifications files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:34:11 +01:00
Stefan Schantl
e246285af4 ids-functions.pl: Add private function to obtain the sid and rev of a
rule.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:33:47 +01:00
Stefan Schantl
e0eb5bc737 ids-functions.pl: Add get_providers_mode() function.
This function is used to gather the modes of the configured providers
and return them as hash.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:33:28 +01:00
Stefan Schantl
65e3aef583 ids-functionsn.pl: Remove logging calls when checking free diskspace.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:17:06 +01:00
Stefan Schantl
52a557a848 ids-functions.pl: Remove logging calls from downloader.
The download script should not directly do the logging stuff.

It simply should download the files for the requested provider and
return an error code on fail.

The logging should be done at another place.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-26 11:14:40 +01:00
Stefan Schantl
e26edcc1c7 ids-functions.pl: Provide better return codes, if the downloader fails.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-25 06:03:40 +01:00
Stefan Schantl
0f2c5211f6 ids-functions.pl: Limit downloader to only one provider.
Remove the option and required code to download the rulesets
for all configured and enabled providers by just calling the downloader
function.

This cause a lot of troubles and if required, directly should be handled
by the processing script.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 21:23:31 +01:00
Stefan Schantl
235e3e92a3 ids-functions.pl: Add get_subscription_code() function.
This function can be used to obtain the subscription code of a given
configured provider.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 21:17:08 +01:00
Stefan Schantl
061391e776 ids-functions.pl: Use If-Modified-Since header to reduce file downloads.
When using the "If-Modified-Since" header, the server can be requested
if a modified version of the file can be served.

In case that is true, the file will be sent and stored by the downloader
function. If the file has not been touched since the last time, the
server will respond with the code "304" (Not modified).

This tells us, that the current stored file is the latest one (still up-to-date)
and we safely can skip the download attempt for this provider.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:51:56 +01:00
Stefan Schantl
f264adda35 ids-functions.pl: Re-order download request handler creation.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:29:21 +01:00
Stefan Schantl
7d8956083b ids-functions.pl: Early load required perl modules.
This will help us to determine if all required perl modules and their
dependencies are avail and load-able.

It also prevents us from doubble loading modules and makes development
and maintainance more easy.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:18:58 +01:00
Stefan Schantl
304ce130fd ids-functions.pl: Remove temporary file, if the download failed.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:04:30 +01:00
Stefan Schantl
e71804fb82 ids-functions.pl: Allow "3" download attempts for each provider before fail.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:04:03 +01:00
Stefan Schantl
de1199e2a3 ids-functions.pl: Drop downloader code for sourcefire based ruleset.
Even if the servers do not support HEAD requests, the remote filesize
(content_length) can be obtained from the connection headers.

This generic method works for all servers and therefore we do not need
the code for handle sourcefire servers in a different way anymore.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-24 20:01:30 +01:00
Stefan Schantl
10beae64bf ids-functions.pl: Skip deleted.rules files
These rulefiles are used by various providers as a kind of reference and
to store rules which have been taken out for correctness, performance
reasons or because of other reasons.

Fixes #12794.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
2022-03-15 17:52:27 +00:00
Stefan Schantl
f349c960e4 ids-functions.pl: Do not longer extract all rulefiles in archive.
Only extract rulefiles which are located in a rules directory and/or in the archive
root.

This prevents us from extracting experimental or binary rules etc. which
often are located in corresponding sub-directories.

Reference: #12794.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Signed-off-by: Michael Tremer <michael.tremer@ipfire.org>
2022-03-14 15:45:00 +00:00
Stefan Schantl
cad8657532 ids-functions.pl: Do not create an empty ignored settings file.
The file will be created by the WUI, when adding the first host.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-05 19:28:24 +00:00
Stefan Schantl
2b12a010d7 ids-functions.pl: Merge same named rulefiles during extract.
In case a rulestarball contains several same-named rulefiles
they have been overwritten each time and so only contained the content
from the last extracted one.

Now the content of those files will be merged by appending the content
to the first extracted one for each time.

Fixes #12792.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-05 19:28:08 +00:00
Stefan Schantl
8353e28ad2 ids-functions.pl: Do not try to chown files while extracting them.
We are almost running as an unprivileged user and therfore have not
the permissions to do this.

This will save us a lot of confusion error messages.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-03-05 14:28:36 +00:00
Stefan Schantl
954dbdadcf ids-functions.pl: Adjust code to deal with new LWP::UserAgent.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
Reviewed-by: Peter Müller <peter.mueller@ipfire.org>
2022-02-09 20:08:45 +00:00
Stefan Schantl
2f62fd0007 ids-functions.pl: Create default rules file file.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-01-08 16:25:17 +01:00
Stefan Schantl
e498947d3a ids-functions.pl: Log the download attempt of a ruleset.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-01-02 09:55:59 +01:00
Stefan Schantl
1b71e2b3c6 Revert "ids-functions.pl: Check if the generated stored rulesfile of a provider"
This commit causes massive troubles in the downloader and therefore
needs to be reverted.

This reverts commit 577e330495.
2022-01-02 09:52:02 +01:00
Stefan Schantl
cad087c74e ids-functions.pl: Check if given filename exists bevore call stat on it.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-01-02 09:49:37 +01:00
Stefan Schantl
dffce1e270 ids-functions.pl: Proper return flase if the downloaded rulestarball
cannot be stored.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2022-01-02 09:48:55 +01:00
Stefan Schantl
ec418b7a08 ids-functions.pl: Drop accidently commited debug output.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 15:46:05 +01:00
Stefan Schantl
c1ccae1ce3 ids-functions.pl: Set bypass flag for whitelisted hosts.
When adding a host to the whitelist set the bypass flag to
immediate take the load from the IDS.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 15:41:43 +01:00
Stefan Schantl
43d12991d1 ids-functions.pl: Dynamically generate file of default suricata rules.
The "/var/ipfire/suricata/suricata-default-rules.yaml" file, now
dynamicall will be generated, based on the enabled application layer
protocols.

Only existing rulefiles for enabled app layer protocols will be loaded.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 15:37:16 +01:00
Stefan Schantl
bb39fac437 ids-functions.pl: Add get_suricata_enable_app_layer_protos().
This function call suricata to obtain a list of enabled application
layer protocols (application/protocol parsers).

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 15:32:52 +01:00
Stefan Schantl
7ccea46172 ids-functions.pl: Remove config files when cleaning up the rules
directory.

If there are one, they safly can be removed because the *.config files
now live in a different folder.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:32:21 +01:00
Stefan Schantl
6983a96eff ids-functions.pl: Adjust classification file for new path.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:31:53 +01:00
Stefan Schantl
c68bcbb298 ids-functions.pl: Do not call stat if no file has been given.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:25:46 +01:00
Stefan Schantl
577e330495 ids-functions.pl: Check if the generated stored rulesfile of a provider
exists before returning the filename.

This will prevent from using and processing non existing files.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:25:26 +01:00
Stefan Schantl
057e895351 ids-functions.pl: Proper return N/A if no ruleset date could be
determined.

If no timestamp could be grabbed for rulestarball of a given provider,
return N/A.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:23:45 +01:00
Stefan Schantl
fc685a36c5 ids-functions.pl: Return N/A if not date for a ruleset could be
determined.

Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:23:45 +01:00
Stefan Schantl
a081f20390 ids-functions.pl: Fix writing for used provider rulefiles.
Signed-off-by: Stefan Schantl <stefan.schantl@ipfire.org>
2021-12-19 13:23:44 +01:00