As far as I know it is not possible directly to bind a condition on the admin state of a port.
But what you can do is something else.
Build a view that shows all ports that are admin down. This should be easy with the filter options.
The result view you use as a datasource to assign a service label to all ports from this view.
This service label will be use by a rule as condition that all ports with this service label should ignore speed.
The tool data2label from @r.sander is only for host labels but i think that it can help also to accomplish the service label task.