compatibility with opnsense

[travisghansen]

How feasible is it to make the integration work with opnsense?

[travisghansen]

Initial conversation here: https://community.home-assistant.io/t/pfsense-integration/349746/21

[travisghansen]

The opnsense api has removed much the the xmlrpc sync functionality but it's feasible to re-add some via a package. Otherwise an entirely new integration could be written using the REST api of opnsense.

[alexdelprete]

Thanks for opening this discussion Travis.

What are your thoughts, is it better using the REST API, like OPNsense developement direction seems to head to, or reimplement XmlRpc via package? What are the pros/cons of the two approaches?

I was wondering if this project could be helpful to achieve the goal: https://github.com/mtreinish/pyopnsense

[travisghansen]

I'm guessing for longevity it would probably be best to use the REST api, but if something could be put together quickly using xmlrpc + pkg it may make sense. At this point the projects seem to have diverged enough I'm really not sure how much can be shared honestly :(

[alexdelprete]

Don't make me lose hope of having a proper integration for OPNsense...:(

If we stick on xmlrpc, that makes more sense in terms of sharing the project, you're afraid it could brake at a certain point because they remove it completely? I know they use it for High Availability Sync, so I don't expect they remove it...

[travisghansen]

@fichtner thanks for the willingness to look at it jointly! When I originally looked at using opnsense I looked at the xmlrpc api because if I want to migrate I must have a replacement for another tool I wrote (https://github.com/travisghansen/kubernetes-pfsense-controller). I was hoping to make that project ‘just work’ when switching over but it became apparent pretty quickly that things had changed enough rendering it not feasible. I would be happy to revisit the whole thing (both hass and kubernetes).

When I reviewed it before it seemed like it was missing either backup_config_section restore_config_section and then also exec_php and merge_installedpackages_section.

Obviously if exec_php is brought back then I can pretty much do anything I need/want.

You can see exactly how I use the api looking at these 3 files:

• https://github.com/travisghansen/hass-pfsense/blob/main/custom_components/pfsense/pypfsense/__init__.py
• https://github.com/travisghansen/kubernetes-pfsense-controller/blob/master/src/KubernetesPfSenseController/Plugin/PfSenseConfigBlock.php
• https://github.com/travisghansen/kubernetes-pfsense-controller/blob/master/src/KubernetesPfSenseController/Plugin/PfSenseAbstract.php

[travisghansen]

OK, did a bit more research and indeed functions have been renamed/etc/etc almost certainly warranting an entire fork of the hass/k8s integrations vs trying to keep them unified.

I was able to readd the exec_php method and exec_shell methods and run the same kind of code I do on the pfSense side. The other major missing method is backup_config_section to be able to retrieve the config (I can achieve this with exec_php alone if absolutely necessary).

In short, if those methods were added back (I don't really want to require extra packages to be installed) I would pretty heavily consider building both integrations to work with OPNsense.

[alexdelprete]

The other major missing method is backup_config_section to be able to retrieve the config (I can achieve this with exec_php alone if absolutely necessary).

There's a tiny plugin to backup/download config.xml called os-api-backup, you can find its source in the OPNsense plugin repository, specifically here. BackupController.php retrieves the config, I hope you can use that code via exec_php.

I think it's also used by the API, endpoint would be: http://opnsense/api/backup/backup/download

if those methods were added back (I don't really want to require extra packages to be installed) I would pretty heavily consider building both integrations to work with OPNsense

I hope it's not a difficult task adding back those methods. Could you make a quick list of what would be exactly needed so we can see if Franco is willing to help on those?

Thank you so much for this Travis, I hope you can get this done with the support of Franco.

[travisghansen]

That package wouldn't be required. Implementing exec_php is as simple as this (it likely would be in the legacy.inc file)..

<?php
// root@OPNsense:/usr/local/etc/inc # cat xmlrpc/hass.inc
function xmlrpc_publishable_hass()
{
    return array("exec_php", "exec_shell");
}

function exec_php($code) {
	eval($code);
	if ($toreturn) {
		return $toreturn;
	}
	return true;
}

function exec_shell($code) {
	mwexec($code);
	return true;
}

Using exec_php it's as simple as this to get the whole config (I have this as working code)..

    def get_config(self):
        script = """
global $config;

$toreturn = [
  "data" => $config,
];
"""
        response = self._exec_php(script)
        return response["data"]

But I would probably re-introduce the backup_config_section method vs forcing someone to use exec_php for this particular purpose. It's a simple task, the bigger question is will it be allowed or not.

In short everything required to support the approach is probably < 30 lines of code added to OPNsense.

[alexdelprete]

In short everything required to support the approach is probably < 30 lines of code added to OPNsense

I hope Franco @fichtner can confirm, and tell us what's the best path to have it approved (PR?).

[alexdelprete]

@fichtner: Franco, sorry to bother you, can you give us a feedback on what Travis proposed when you can? Much appreciated, thank you. :)

[fichtner]

Sorry for the delay. We went over the situation internally and here are our thoughts. For xmlrpc unused functions were scrubbed a long time ago, especially such things that could exfiltrate (configuration) data or execute code which is what seems to be missing for proper support.

With that being said xmlrpc has a drop in directory that can hold plugin-based extensions for the missing methods for example, but for previous reason an inclusion of two questionable functions into our direct ecosystem is unlikely.

The other issue is that apart from that situation the queries done in php need to be ported or adapted. In some cases I'm not sure you can grab the relevant data. I think this is where more work will have to be done anyway.

Since something has to be dropped into an OPNsense installation anyway it might make sense to move the queries to the server and have xmlrpc methods for returning the whole data so that nothing has to be done in RPC style and it's a bit easier to do maintenance and versioning on internal and external changes especially across major version changes on the OPNsense side.

Long term xmlrpc is a candidate for replacement. Most of our central management already lives on top of our APIs and it's less friction in general plus it doesn't require an external xmlrpc library that is not well-maintained in the first place.

Cheers,
Franco

[travisghansen]

Yeah with the functions hacked back into the xmlrpc system.

[alexdelprete]

but if I understood correctly, Franco said they were not keen to reintroduce them back, or you did it like he suggested, described in point 2 of my previous post?

[travisghansen]

I leveraged the ‘plugin’ system mentioned (the file I shared a week ago). Basically 1 extra file has to be installed on the fw machine and then I’ve updated all the other calls with various adjustments to get the data I want to make the integration work.

[alexdelprete]

And the file on the server should be installed through the OPNsense plugin system?

You were able to maintain the interfaces in the plugin so you don't have to modify the HA component?

[travisghansen]

I’m not sure the appropriate way to install the file, perhaps the plugin system is the best option.

Much of the integration is the same but it required substantial changes as well to make it compatible with opnsense.

[alexdelprete]

Travis, I really appreciate your efforts, and all OPNsense+HA users will be really happy about this. A unified pfsense/opnsense component. That is awesome. :)

Can I test it? I know it still PoC but no problems, I'm used to beta testing...

[mimugmail]

If there's a need to add code to core files its getting a bit tricky to build a plugin

[alexdelprete]

If there's a need to add code to core files its getting a bit tricky to build a plugin

It's an xmlrpc plugin, there should be a specific folder for those. Franco mentioned it in his previous post:

xmlrpc has a drop in directory that can hold plugin-based extensions for the missing methods for example, but for previous reason an inclusion of two questionable functions into our direct ecosystem is unlikely.

Do you see problems for that?

[travisghansen]

It will not be needed to alter an existing file. The xmlrpc ‘plugin’ system allows us to simply drop a new/additional file in place and it will get picked up.

[travisghansen]

Give it a trial: https://github.com/travisghansen/hass-opnsense

Let's not share it widely for now as I'm still unclear where it'll go.

[alexdelprete]

If there's a need to add code to core files its getting a bit tricky to build a plugin

@mimugmail Michael, I just installed an alpha version of the integration, and it's working fine, finally we have OPNsense integrated with Home Assistant thanks to Travis.

As per instructions, here's what's needed to install on OPNsense:

sh
cat << 'EOF' > /usr/local/etc/inc/xmlrpc/hass.inc
<?php
function xmlrpc_publishable_hass()
{
    return array(
        "exec_php_xmlrpc",
        "exec_shell_xmlrpc"
    );
}

function exec_php_xmlrpc($code)
{
    eval($code);
    if ($toreturn)
    {
        return $toreturn;
    }
    return true;
}

function exec_shell_xmlrpc($code)
{
    mwexec($code);
    return true;
}

EOF
chown root:wheel /usr/local/etc/inc/xmlrpc/hass.inc
chmod 644 /usr/local/etc/inc/xmlrpc/hass.inc

Question: Is it possible to build a plugin that executes those actions when installed? It would help new users by making the installation very easy using your repo.

Thanks a lot for your help,

Alessandro

[alexdelprete]

Great job!!! :)

Works. But there's a problem for interface traffic stats:

[travisghansen]

Those look great! That’s one of the things on my list is to add some pretty pictures to the README

[alexdelprete]

We can put pics and also share the code...I'm not very good on UI stuff generally, but luckily there is ApexCharts custom component that does much of the magic. In those you see ApexCharts for the graph and uptime-card for the first two.

When you're ready to publish, let me know and I'll send you the code and the pics.

[alexdelprete]

Travis, looking at this code, the status of the gateway can be: force_down, down, loss, delay, none (that you remapped to online).

Is that correct? I'm trying to determine when the interface is really down or up. From what I see, it's down when the status is "down" or "force_down" and it is up when status is "none". Is that a proper assumption?

I was thinking if it wouldn't be better to use the same terminology of pfsense/opnsense for the status: up/down, Iinstead of online/offline. It would be good to have a binary_sensor for the connection state too. Let me know your thoughts...

It's working good, I updated to last version.

BTW: I'm posting here because there's no discussions tab in the hass-opnsense repo, if you create it I'll post over there. :)

Thanks.

                          $status = $gateways_status[$gateway['name']];
                          if (stristr($status['status'], 'force_down')) {
                              $online = gettext('Offline (forced)');
                              $gateway_label_class = 'danger';
                          } elseif (stristr($status['status'], 'down')) {
                              $online = gettext('Offline');
                              $gateway_label_class = 'danger';
                          } elseif (stristr($status['status'], 'loss')) {
                              $online = gettext('Warning (packetloss)');
                              $gateway_label_class = 'warning';
                          } elseif (stristr($status['status'], 'delay')) {
                              $online = gettext('Warning (latency)');
                              $gateway_label_class = 'warning';
                          } elseif ($status['status'] == 'none') {
                              $online = gettext('Online');
                              $gateway_label_class = 'success';
                          }

[travisghansen]

I’ve enabled discussions on the other project. I would probably defer the binary sensor as a template sensor and let folks create those with whatever logic desired.

[alexdelprete]

Ok for the binary sensor, regarding the terminology: up/down (used by pfsense/opnsense) vs online/offline?

I'll write next posts in the other repo. :)

[alexdelprete]

Also the gateway info doesn't seem correct:

this is opnsense:

[travisghansen]

The telemetry logging should give some insight to this as well.

[alexdelprete]

Finally I see something regarding the gateway, apart the status. For the interfaces' traffic stats, no change.

[travisghansen]

Send over the debug logs and we'll see if it helps understand why you aren't seeing interface stats..

[alexdelprete]

Send over the debug logs and we'll see if it helps understand why you aren't seeing interface stats..

Also the Gateway status doesn't work. Should be "online".

[mimugmail]

I updated the community repo https://github.com/mimugmail/opn-repo
If you have the repo installed you should find a plugin os-homeassistant-maxit which will place the file and set permissions

[alexdelprete]

Michael, thank you so much for this, we really appreciate your help. I just installed it through OPNsense plugin UI and it backed up my existing hass.inc as hass.inc.pkgsave. I compared the two files as there was a 2 bytes differnce, it was the cr/lf at at end of the file. But it's working fine. :)

Could you put a link to the hass-opnsense repo in the info of the plugin?

@travisghansen we have the plugin. can we add this method of installation to the readme? It'll be much easier for users. :)

[travisghansen]

Yeah let’s add it to the readme. I’m going to be pretty tied up this week so won’t have a lot time however.

[alexdelprete]

If you want I can update the readme...do I have to clone it locally or can I edit it online via GH?

[travisghansen]

You’ll probably need to ‘fork’ it and submit a PR.

[alexdelprete]

ok, will do...so you'll only need to accept the PR.