Extend the authlog / secure log plugin #342

damasch · 2023-08-02T09:52:00Z

Extend the authlog / secure log plugin

Extend code for Issue fox-it#286

cecinestpasunepipe · 2023-08-09T16:15:44Z

dissect/target/plugins/os/unix/log/auth.py

 _TS_REGEX = r"^[A-Za-z]{3}\s*[0-9]{1,2}\s[0-9]{1,2}:[0-9]{2}:[0-9]{2}"
 RE_TS = re.compile(_TS_REGEX)
 RE_TS_AND_HOSTNAME = re.compile(_TS_REGEX + r"\s\S+\s")


+# New regex pattern for getting the log entries


Can you please add some tests for these?

I will add some tests for this scenario.
Should i place this tests into tests/data/unix/logs/auth/auth.log (and zips) and tests/test_plugins_os_unix_log_auth.py.

for the test data: tests/data/plugins/os/unix/log/auth/auth.log would make more sense on how it is currently structured in the tests.
The test file path is fine :)

I see no tests have been added yet. Just to confirm tests/data/plugins/os/unix/auth/auth.log is a perfect location for the test data.

dissect/target/plugins/os/unix/log/auth.py

Miauwkeru · 2023-08-16T12:24:54Z

dissect/target/plugins/os/unix/log/auth.py

+ data.update(self.apply_regex_on_message(RE_SSH_ACCEPTED_PASSWORD, log_entry["message"]))
+ data.update(self.apply_regex_on_message(RE_SSH_ACCEPTED_PUBLICKEY, log_entry["message"]))
+ data.update(self.apply_regex_on_message(RE_SSH_PAM_UNIX, log_entry["message"]))
+ data.update(self.apply_regex_on_message(RE_SSH_CONNECTION, log_entry["message"]))
+ data.update(self.apply_regex_on_message(RE_SUDO_COMMAND, log_entry["message"]))
+ data.update(self.apply_regex_on_message(RE_CRON_PAM_UNIX, log_entry["message"]))


Instead of always doing reg expressions on the log_entry, would it not be better if you did some short per-processing first?

e.g. Check if the message starts with some unique identifiers.

Basically create a selector for the regex you want.

I've try this with a loop and a break. It is not ideal but bettern than before. I will think about the identifier solution. But with the "pam unix" entries it will be a little bit difficult.

dissect/target/plugins/os/unix/log/auth.py

Miauwkeru · 2023-08-16T12:30:12Z

dissect/target/plugins/os/unix/log/auth.py

+
+# New regex pattern for interpreting the SSH message "Accepted password"
+# Mar 29 10:43:01 my_unix_host sshd[1193]: Accepted password for test_user from 127.0.0.1 port 52942 ssh2
+RE_SSH_ACCEPTED_PASSWORD = re.compile(r'^Accepted\spassword\sfor\s(?P<user>[\S\_]+)\sfrom\s(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$')


I see some duplicate parts inside your regular expressions. Could you make it more concise and consistent?

For example I do see that the <user> information is used in a lot of regular expressions, however they are all different, it might be nice to make them consistent.

In Accepted password and Accepted Publickey, both messages handle the same user. If I pump the data into a eks or somthing else, the keys could be filtered on all entries/entities. So i can match the User activity to other data

I mean more, that in this case you have made a lot of duplicate regexes. Another example would be the ip addresses regex that are duplicated everywhere.

It might also be an idea to split the regex over multiple lines. So instead of:

re.compile('^Accepted\spassword\sfor\s(?P<user>[\S\_]+)\sfrom\s(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$')

you can do:

re.compile( '^Accepted\spassword\sfor\s' # Check for correct preamble '(?P<user>[\S\_]+)\sfrom\s' # Retrieve user '(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})' '\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$' )

which makes it a bit easier to read and comment too

Miauwkeru

@damasch My apologies for the very long wait, I will pick this up right now and give a more in depth review today.
In addition I will do some more local tests with different distributions to see whether it works well.

Miauwkeru · 2023-10-06T07:48:55Z

dissect/target/plugins/os/unix/log/auth.py

+
+# New regex pattern for interpreting the SSH message "Accepted password"
+# Mar 29 10:43:01 my_unix_host sshd[1193]: Accepted password for test_user from 127.0.0.1 port 52942 ssh2
+RE_SSH_ACCEPTED_PASSWORD = re.compile(r'^Accepted\spassword\sfor\s(?P<user>[\S\_]+)\sfrom\s(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$')


I mean more, that in this case you have made a lot of duplicate regexes. Another example would be the ip addresses regex that are duplicated everywhere.

It might also be an idea to split the regex over multiple lines. So instead of:

re.compile('^Accepted\spassword\sfor\s(?P<user>[\S\_]+)\sfrom\s(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$')

you can do:

re.compile( '^Accepted\spassword\sfor\s' # Check for correct preamble '(?P<user>[\S\_]+)\sfrom\s' # Retrieve user '(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})' '\sport\s(?P<port>[0-9]{1,5})\s(?P<authservice>\w+)$' )

which makes it a bit easier to read and comment too

Miauwkeru · 2023-10-06T07:52:23Z

dissect/target/plugins/os/unix/log/auth.py

 _TS_REGEX = r"^[A-Za-z]{3}\s*[0-9]{1,2}\s[0-9]{1,2}:[0-9]{2}:[0-9]{2}"
 RE_TS = re.compile(_TS_REGEX)
 RE_TS_AND_HOSTNAME = re.compile(_TS_REGEX + r"\s\S+\s")


+# New regex pattern for getting the log entries


for the test data: tests/data/plugins/os/unix/log/auth/auth.log would make more sense on how it is currently structured in the tests.
The test file path is fine :)

Miauwkeru

So I see some redundant definitions, for example, all the PAM regexes are pretty much the same. (besides the service it starts with) So i recommend unifying that information.

It might also be an idea to try and detect what you are working with dependent on:
service and the start of the message so you can select which regex to use that way. Instead of trying them all out.

But before doing all of that, I would recommend adding your tests first, so it becomes easier to refactor for you :)

Miauwkeru · 2023-10-06T10:38:18Z

dissect/target/plugins/os/unix/log/auth.py

+ ("string", "userid"),
+ ("string", "useridassociate"),


Suggested change

("string", "userid"),

("string", "useridassociate"),

("string", "uid"),

("string", "associate_uid"),

wouldn't this make it more readable?

Miauwkeru · 2023-10-06T10:51:36Z

dissect/target/plugins/os/unix/log/auth.py

+ 'ts': ts,
+ }
+
+ log_entry = re.match(RE_ENTRY, line)


Instead of a regex that also gets the message, would a line.split(":", 1) not also work here?
As far as I can see, the : is a constant that splits the service and process_id from the message

then it would look a bit like this:

prefix, message = line.split(":", 1) # get service and process_id log_entry = re.match(RE_ENTRY, prefix) ... # Do other stuff with message here

Miauwkeru · 2023-10-06T11:00:18Z

dissect/target/plugins/os/unix/log/auth.py

+
+ for regex in RE_LIST:
+ matches = self._apply_regex_on_message(regex, log_entry["message"])
+ if matches != {}:


You can but the accessing of message before the for loop, so it doesn't do it every iteration

And using the := in if conditions will tell it to only go into that condition if matches is a not None or "empty" such in the case of an empty dict.

Suggested change

for regex in RE_LIST:

matches = self._apply_regex_on_message(regex, log_entry["message"])

if matches != {}:

message = log_entry["message"]

for regex in RE_LIST:

if matches := self._apply_regex_on_message(regex, message):

Miauwkeru · 2023-10-06T11:12:59Z

dissect/target/plugins/os/unix/log/auth.py

+# New regex pattern for interpreting the session message "cron"
+# Mar 29 17:07:25 my_unix_host sshd[4651]: pam_unix(sshd:session): session opened for user ubuntu by (uid=0)
+RE_CRON_PAM_UNIX = re.compile(r'^pam_unix\(cron:(?P<cron>.*)\): +session +(?P<sessionmode>closed|opened)\sfor\suser\s(?P<user>\w+)(?:\(uid=(?P<useridassociate>\w+)\))?(?:\sby\s)?(?:\(uid=(?P<userid>\w+)\))?$')


Is this one not redundant? I see it is very specific for cron, but basically does the same as RE_SSH_PAM_UNIX.

I'd suggest changing it to RE_PAM_UNIX with r"^pam_unix$?(P<auth_service>\w+):(P<service_call>\w+)$:" or something similar.

Then it is more flexible in regards of any future pam_unix information.

Miauwkeru · 2023-10-06T11:16:06Z

dissect/target/plugins/os/unix/log/auth.py

+
+# New regex pattern for interpreting the connection mode on close message "ssh"
+# Mar 29 17:07:19 my_unix_host sshd[4649]: Connection closed by 85.245.107.41 port 54790 [preauth]
+RE_SSH_CONNECTION = re.compile(r'^Connection\s(?P<conectionmode>closed)\sby\s(?P<remoteip>[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}|[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4}:[0-9a-zA-Z]{0,4})\sport\s(?P<port>[0-9]{1,5})$(?P<misc>.*)$')


is $(?P<misc>.*)$ even used? as I recall that $ defines then end of the line if I remember correctly.

Miauwkeru · 2023-10-06T11:20:35Z

dissect/target/plugins/os/unix/log/auth.py

+
+# New regex pattern for interpreting the PAM UNIX message "ssh"
+# Jul 5 13:20:15 test-VirtualBox sudo: pam_unix(sudo:session): session opened for user root(uid=0) by test(uid=0)
+RE_SSH_PAM_UNIX = re.compile(r'pam_unix\((?P<authservice>sshd):.*\): +session +(?P<sessionmode>closed|opened)\sfor\suser\s(?P<user>\w+)(?:\(uid=(?P<useridassociate>\w+)\))?(?:\sby\s)?(?:\(uid=(?P<userid>\w+)\))?$')


same comment about the pam for the ssh here.

cecinestpasunepipe · 2023-10-06T13:09:03Z

dissect/target/plugins/os/unix/log/auth.py

 _TS_REGEX = r"^[A-Za-z]{3}\s*[0-9]{1,2}\s[0-9]{1,2}:[0-9]{2}:[0-9]{2}"
 RE_TS = re.compile(_TS_REGEX)
 RE_TS_AND_HOSTNAME = re.compile(_TS_REGEX + r"\s\S+\s")


+# New regex pattern for getting the log entries


I see no tests have been added yet. Just to confirm tests/data/plugins/os/unix/auth/auth.log is a perfect location for the test data.

damasch · 2023-11-29T09:42:08Z

It's been a infinity since I can work for this project. I have to apologize for this very long delay.

Are you interested in a remote web talk about this request?

Maybe we can organize a talk about this.

It is ok for you?

Thanks

DissectBot · 2023-11-30T10:12:29Z

@damasch thank you for your contribution! As this is your first code contribution, please read the following Contributor License Agreement (CLA). If you agree with the CLA, please reply with the following information:

@DissectBot agree [company="{your company}"]

Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement ("Agreement") governs your Contribution(s) (as defined below) and conveys certain license rights to Fox-IT B.V. ("Fox-IT") for your Contribution(s) to Fox-IT"s open source Dissect project. This Agreement covers any and all Contributions that you ("You" or "Your"), now or in the future, Submit (as defined below) to this project. This Agreement is between Fox-IT B.V. and You and takes effect when you click an “I Accept” button, check box presented with these terms, otherwise accept these terms or, if earlier, when You Submit a Contribution.

Definitions.
"Contribution" means any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to Fox-IT for inclusion in, or documentation of, any of the software products owned or managed by, or on behalf of, Fox-IT as part of the Project (the "Work").
"Project" means any of the projects owned or managed by Fox-IT and offered under a license approved by the Open Source Initiative (www.opensource.org).
"Submit" means any form of electronic, verbal, or written communication sent to Fox-IT or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Fox-IT for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to Fox-IT and to recipients of software distributed by Fox-IT a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works.
Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to Fox-IT and to recipients of software distributed by Fox-IT a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, maintain, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed.
Representations. You represent that:
- You are legally entitled to grant the above license.
- each of Your Contributions is Your original creation (see section 8 for submissions on behalf of others).
- Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions.
Employer. If Your Contribution is made in the course of Your work for an employer or Your employer has intellectual property rights in Your Submission by contract or applicable law, You must secure permission from Your employer to make the Contribution before signing this Agreement. In that case, the term "You" in this Agreement will refer to You and the employer collectively. If You change employers in the future and desire to Submit additional Contribution for the new employer, then You agree to sign a new Agreement and secure permission from the new employer before Submitting those Contributions.
Support. You are not expected to provide support for Your Contribution, unless You choose to do so. Any such support provided to the Project is provided free of charge.
Warranty. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
Third party material. Should You wish to submit work that is not Your original creation, You may only submit it to Fox-IT separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which You are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]".
Notify. You agree to notify Fox-IT of any facts or circumstances of which You become aware that would make the above representations inaccurate in any respect.
Governing law / competent court. This Agreement is governed by the laws of the Netherlands. Any disputes that may arise are resolved by arbitration in accordance with the Arbitration Regulations of the Foundation for the Settlement of Automation Disputes (Stichting Geschillenoplossing Automatisering – SGOA – (www.sgoa.eu), this without prejudice to either party"s right to request preliminary relief in preliminary relief proceedings or arbitral preliminary relief proceedings. Arbitration proceedings take place in Amsterdam, or in any other place designated in the Arbitration Regulations. Arbitration shall take place in English.

Miauwkeru · 2023-11-30T10:17:46Z

@damasch it seems our bot missed you when asking for a CLA.

It's been a infinity since I can work for this project. I have to apologize for this very long delay.

Are you interested in a remote web talk about this request?

Maybe we can organize a talk about this.

It is ok for you?

Thanks

As for your question, we can set something up to elaborate on the suggested changes. If you send us your email on [email protected] we'll schedule a call

JSCU-CNI · 2024-03-11T08:35:42Z

What is the status of this PR?

Schamper · 2024-03-16T13:11:09Z

This PR has been a bit back and forth unfortunately. @damasch, if you're still willing to accept the CLA, we can take it from here and process the suggestions ourselves.

Miauwkeru · 2024-06-24T12:55:53Z

Hello @damasch, we haven't heard a response from you. Could you still sign the CLA?

Otherwise, we will close this pr in 2 weeks due to inactivity.

Schamper · 2024-07-01T09:25:18Z

Hello @damasch, we haven't heard a response from you. Could you still sign the CLA?

Otherwise, we will close this pr in 2 weeks due to inactivity.

And re-implement it ourselves?

Miauwkeru · 2024-07-05T07:55:57Z

Hello @damasch, we haven't heard a response from you. Could you still sign the CLA?
Otherwise, we will close this pr in 2 weeks due to inactivity.

And re-implement it ourselves?

yes

Miauwkeru · 2024-07-05T07:56:11Z

Closed due to inactivity

Extend the authlog / secure log plugin

8cfe461

Extend code for Issue fox-it#286

Schamper requested review from Miauwkeru and cecinestpasunepipe August 2, 2023 16:55

cecinestpasunepipe requested changes Aug 9, 2023

View reviewed changes

Miauwkeru requested changes Aug 16, 2023

View reviewed changes

damasch added 3 commits September 6, 2023 14:24

Merge branch 'fox-it:main' into DEV/Auth-Log-format-update-on-rfc5424

f904ba9

Handle 4 of 7 isseues

536e573

Merge branch 'main' into DEV/Auth-Log-format-update-on-rfc5424

921982c

Schamper requested review from Miauwkeru and cecinestpasunepipe September 19, 2023 11:12

Miauwkeru requested changes Oct 6, 2023

View reviewed changes

cecinestpasunepipe requested changes Oct 6, 2023

View reviewed changes

Miauwkeru closed this Jul 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend the authlog / secure log plugin #342

Extend the authlog / secure log plugin #342

damasch commented Aug 2, 2023

cecinestpasunepipe Aug 9, 2023

damasch Sep 6, 2023

Miauwkeru Oct 6, 2023

cecinestpasunepipe Oct 6, 2023

Miauwkeru Aug 16, 2023

damasch Sep 6, 2023

Miauwkeru Aug 16, 2023

damasch Sep 6, 2023 •

edited

Loading

Miauwkeru Oct 6, 2023

Miauwkeru left a comment

Miauwkeru Oct 6, 2023

Miauwkeru Oct 6, 2023

Miauwkeru left a comment

Miauwkeru Oct 6, 2023 •

edited

Loading

Miauwkeru Oct 6, 2023

Miauwkeru Oct 6, 2023

Miauwkeru Oct 6, 2023 •

edited

Loading

Miauwkeru Oct 6, 2023

Miauwkeru Oct 6, 2023

cecinestpasunepipe Oct 6, 2023

damasch commented Nov 29, 2023

DissectBot commented Nov 30, 2023

Contribution License Agreement

Miauwkeru commented Nov 30, 2023

JSCU-CNI commented Mar 11, 2024

Schamper commented Mar 16, 2024

Miauwkeru commented Jun 24, 2024

Schamper commented Jul 1, 2024

Miauwkeru commented Jul 5, 2024

Miauwkeru commented Jul 5, 2024

Extend the authlog / secure log plugin #342

Extend the authlog / secure log plugin #342

Conversation

damasch commented Aug 2, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damasch Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miauwkeru left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miauwkeru left a comment

Choose a reason for hiding this comment

Miauwkeru Oct 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Miauwkeru Oct 6, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

damasch commented Nov 29, 2023

DissectBot commented Nov 30, 2023

Contribution License Agreement

Miauwkeru commented Nov 30, 2023

JSCU-CNI commented Mar 11, 2024

Schamper commented Mar 16, 2024

Miauwkeru commented Jun 24, 2024

Schamper commented Jul 1, 2024

Miauwkeru commented Jul 5, 2024

Miauwkeru commented Jul 5, 2024

damasch Sep 6, 2023 •

edited

Loading

Miauwkeru Oct 6, 2023 •

edited

Loading

Miauwkeru Oct 6, 2023 •

edited

Loading