Frequent buses - both direct express which use the highway and local which use the old toll-free road - link Port Dickson with Seremban. Travel time is about one hour and cost RM4. On weekdays there only seems to be one company offering buses between Seremban and Port Dickson, one per hour. If coming from other parts of Malaysia, the easiest way by bus will be to travel via Seremban.
Local buses also link Port Dickson with Sungai Pelek in Sepang district in Selangor, where there are connections to the Kuala Lumpur International Airport. To the south, local buses go to Pengkalan Kempas and Tanjung Agas (these buses actually go right up to Kuala Linggi in Malacca) where there are connections to Malacca city.
If coming from Malacca via Seremban by bus, allow about 4-5 hours to get to Port Dickson. It's not worth it.