Making BlueCoat ASG use one Forwarding VIP for multiple TCP Services

Friday, 07 Sep 2018

BlueCoat Advanced Secure Gateway

The BlueCoat ASG are a series of Proxy Appliances, sold by Symantec and commonly used for a combination of Proxy Types, such as:

  • Forward Proxy
    • HTTP/S Web Proxy to Internet
    • SOCKS Proxy to Internet
  • Reverse Proxy
    • ActiveSync Proxy to Microsoft Exchange
    • Oracle OAM Proxy to Oracle Authentication

Once you get past the fact that the Admin Panel runs Java (eww), they aren't bad boxes to configure and use; but if you're coming at these from a Networking background, be warned - everything is split-out into a different section, and not much is where you'd expect it to be.

The Scenario

My Scenario was using the Virtual IP (VIP) functionality, to offload some Reverse Proxy functionality to a downstream Appliance; which sits behind the BlueCoat to allow it to (in future) perform some DPI and other bits and pieces. I needed to create one VIP Address, but have it forward three TCP Ports/Services:

  • SOCKS (Running on TCP/80)
  • SSH (TCP/22)
  • RDP (TCP/3389)

The Proxy VIP (BlueCoat VIP) is 10.99.99.99.

The Downstream Proxy Real IP (BlueCoat Forwards/NATs to) is 10.98.1.2.

Coming from a Networking background, my mind went to NAT, VIPs and Load Balancer Groups; but BlueCoat doesn't play ball that way.

Elements to Configure

The following sections of the Admin Panel/BlueCoat Configuration need to be modified to achieve this:

  • Proxy Services
  • Forwarding Hosts
  • Forwarding Host Groups
  • Visual Policy Manager (Forwarding Layer)

So let's crack on.

Proxy Services

  1. From BlueCoat Admin Panel, navigate to Proxy -> Configuration -> Services -> Proxy Services
  2. Add your "Custom Service Groups" (mine is called "CustomGroup")
  3. Add your SOCKS/SSH/RDP Service Group elements, under the "Custom Service Groups" section
  4. Set these to "Intercept", and Source to "All" (or specific Downstream User Subnets, which should be able to access the VIP)
  5. Click "Apply" (bottom-right) when done

undefined

Forwarding Hosts

  1. From BlueCoat Admin Panel, navigate to Proxy -> Configuration -> Forwarding-> Forwarding Hosts
  2. Add your "Forwarding Hosts", one per TCP Service you want to listen to (I have three)
  3. Within each Forwarding Host, set the "Type" to "Server", and specify the relevant "TCP:" port you want to use
  4. Click "Apply" (bottom-right) when done

undefined

Forwarding Hosts Groups

  1. From BlueCoat Admin Panel, navigate to Proxy -> Configuration -> Forwarding-> Forwarding Hosts -> Forwarding Groups
  2. Define a custom Group Alias (mine is "Custom-Group") to use as the Group container for the three Forwarding Hosts you input
    1. undefined
  3. Add your Custom Hosts to the Custom Group:
    1. undefined

Visual Policy Manager (Forwarding Layer)

  1. From BlueCoat Admin Panel, navigate to Proxy -> Configuration -> Policy -> Visual Policy Manager -> Launch
  2. Within "Forwarding Layer", add an entry using your VIP Address (10.99.99.99), with a colon (:) after, and "Any" Service
    1. undefined
  3. Tie this to a new Action called "Action-CustomGroup"
    1. undefined
  4. Edit "Action-CustomGroup" to relate to the Forwarding Host Group you specified earlier (Custom-Group), and "Deny the request" if the Forwarding Host (i.e. all TCP Ports within that Group) aren't available
    1. undefined
  5. Within VPM, apply the policy to the BlueCoat ASG - navigate to File -> Install Policy on SG Appliance
  6. Party on down, we're done!

Wrapping it up

BlueCoat ASG will quite happily let you specify a collection of individual Forwarding Hosts, each with an individual TCP Port; and the VPM will then let you specify these multiple individual Forwarding Hosts - but it won't actually VIP Forward the traffic, and will fail miserably.

The above seems to be the "BlueCoat Way" of doing this, and definitely works.

When BGP SoO Site of Origin goes the wrong way

Sunday, 12 Aug 2018

BGP Site of Origin (SoO)

I have a scenario where an "internal" Service Provider (SP) MPLS Network interfaces with a Third Party's MPLS Network, as an IPVPN - rather than a true MP-BGP Handoff; or in other words, "I happen to know it's underpinned by MPLS so I'll call it that, even though technically it's not MPLS Presentation to me" (the same way most Enterprise Network shops refer to their WAN as "MPLS").

Unlike the base assumption of most Cisco articles on SoO, I don't actually control the Provider Edge (PE) Routers on this Third Party (let's say, "BT") MPLS Network; and nor am I the Third Party themselves. What I'd like to do is identify Prefixes I have on my IPVPN "Overlay" MPLS Network, from CE Routers on said IPVPN Overlay Network that I do control, and block them from coming back into my own SP MPLS Network. I thought BGP Site of Origin (SoO) might be my friend here...

The Scenario Topology

undefined

This is the Scenario Network Topology, showing:

  • 2x My MPLS SP Network Provider Edge (PE) Routers
  • 1x My MPLS SP Network Route Reflector (RR) Router
  • 2x BT MPLS PE Routers (Location Unknown)
  • 2x eBGP Peerings from My <-> BT MPLS PE Routers
  • 2x Repeated BGP SoO Communities (65432:999), applied to VRF (IPVPN) "BLAH"

What I thought would happen

Given the BGP SoO Attribute is applied towards the BT (Third Party MPLS Network) PE Router, I thought I'd be able to jump on one of my MPLS Enterprise Network CE Routers, and see the SoO Attribute 65432:999 applied, as it made sense to me that this configuration would "advertise" the BGP SoO Extended Community from Me -> BT:

PE1#
router bgp 65432
 address-family ipv4 vrf BLAH
  neighbor 192.168.0.1 remote-as 2856
  neighbor 192.168.0.1 send-community both
  neighbor 192.168.0.1 soo 65432:999

Where I then duly hop onto my CE1 Router and issue the following, expecting to see the SoO Tag for my VRF "BLAH" 99.99.99.99/32 Network:

undefined

But, no dice - it's just a normal IPv4 BGP Prefix with no Extended Communities. What gives?

What actually happens

At this point, confused, I start to wonder if I'm misunderstanding what SoO does and is, and getting confused between the simplistic Cisco and Internet examples of SoO (which are aimed at Service Providers, from their perspective, towards a singular Customer Edge/Customer) - so I poke around. The first point I poke around on is a "non-SoO Tagging PE Router', PE99 - which has an attachment to VRF "BLAH" (an ingress/attachment point into My MPLS Network, for VRF "BLAH"; but performs no SoO tagging) - and see what I see:

PE99#sh ip bgp vpnv4 vrf VLAH 99.99.99.99/32
<snip>
99.99.99.99/32
  ...Extended Community: SoO:65432:999 RT...

Which starts to make it click - the SoO must be applied "the wrong-way-around" from what I thought, and be an "ingress only" behaviour, as otherwise I wouldn't see it this side of the MPLS CE-PE Network Handoff fence. Or more succinctly, this is the direction/Routing Domain the SoO Tag is applied into:

undefined

Even though I first looked at this part of the SoO command as an "advertise-out"/egress behaviour; it's actually a "Match Packets from this BT Peer, mark them with this when advertised deeper back to us" behaviour. Because I looked at this part of the config as an "advertise SoO to neighbour" behaviour:

PE1#
  neighbor 192.168.0.1 soo 65432:999

Which is different to what I first expected ("advertise-out" behaviour), which would have been this:

undefined

What have I learned?

In effect, then, depending on your "directional thinking", based on the Cisco IOS syntax, you might be unpleasantly surprised by how this works - to my mind, it's working the wrong-way-around from what the config syntax would suggest. What actually happens, as a result of just one line of config, is:

  1. On PE1# (My<->BT Network 1st CE-PE Handoff)
    1. Apply SoO Tag 65432:999 inbound/ingress (BT->Me) for all BT-side Prefixes
    2. Advertise this SoO Tag 65432:999 deeper back to My SP Network (not BT's at all)
  2. On PE99# (Any other CE-like Router on My Network; Attachment Point into VRF "BLAH")
    1. See SoO Tag 65432:999 on BT-native Prefixes
    2. Do nothing about it; "advertise on" SoO Prefix (don't strip it out on re-advertise to another PE/Router)
  3. On PE2# (My<->BT Network 2nd CE-PE Handoff)
    1. See SoO Tag 65432:999 pre-egress/outbound (just before Me->BT) for the BT-side Prefix
    2. Because the Me<->BT eBGP Peer has the same SoO Tag set, don't allow it out to BT Router ("reverse behaviour")

The same would then happen for BT->PE2->My MPLS->PE1, performing an overlapping-behaviour for the secondary/dual-homed path between My Network<->BT's Network.

So these bits are wrong then

Which means, given SoO is an "inbound behaviour", not an "outbound" behaviour, the whole concept of tagging these with 65432:999 as the SoO Tag doesn't make sense; it probably should be 2856:999, to show these are BT-native, not My Network-native.

It also means I should re-think why I'm using SoO here, as a Tag+Block/Route Map technique, using bog-standard BGP Communities, might be a better fit for the behaviour I wanted.

I've been here before

Sadly, I've fallen victim to this presumed "outbound behaviour of the config" before with my friend BGP AS-Override, which also has a strange "not the way you might expect" behaviour, but that's one for another blog post. Key points here are:

  • Trust nothing
  • Lab everything
  • Assume that Cisco Support Forums write-up that looked exactly like your scenario was too good to be true

The difference between BGP RD and RT

Friday, 10 Aug 2018

BGP Route Descriptor and Route Target

Let me caveat this post by saying I'm not a Service Provider (SP) kid by trade; I spend my life doing Enterprise, Data Centre and Wireless - so all this MPLSery is new territory for me, and my imaginary sidekick-dog friend ("Hi Jake!") - which means this might be technically incorrect, but this is how the concepts of RD and RT finally "clicked" for me.

How it was explained to me

undefined

When I first starting Googling for Dear Life (TM) about this (because I needed to spin up a new VRF/IPVPN/L3VPN on our MPLS Network), and looked at a few existing config excerpts, I thought they were both the same thing, which seems valid:

vrf definition ADVENTURE-TIME-VRF
 rd 192.168.0.1:999
 route-target export 65432:999
 route-target import 65432:999

I didn't really question the fact that the Export/Import Route Target (RT) was the same (and didn't know about "Full Mesh VRF" vs "Hub-and-Spoke VRF"), but it did strike me as odd that the RD wasn't the same as the RT, given all the explanation I'd read said things like:

The RD is used to keep all prefixes in the BGP table unique between Customers or VRFs...

Which I read thinking:

"Hmm, that makes sense; BGP will just append the RD in-front of the Prefix, to identify the VRF it belongs to. But wouldn't that mean the RD should be the same for each PE Router, the same for each instantiation of that VRF/Customer across the network?"

So then why the differing RD from the RTs?

Why bother with the extra admin work of creating a different value each time, between the RD and RT?

How I now understand it

undefined

When I started exploring Full Mesh VRF vs Hub-and-Spoke VRF, it started to click into place - the RT and RD aren't really related, and I think there's some missing text from the common definition of how RD's are enacted:

  • RD = Route(r) Descriptor
  • RT = Rout(ing Table) Target

When I looked around the configs we had elsewhere, the pattern become clear; it decomposed like this:

vrf definition <VRF Human-friendly Name>
 rd <Router Loopback0>:<VRF RT No>
 route-target export <Router ASN>:<VRF RT No>
 route-target import <Router ASN>:<VRF RT No>

It's starting to click

Then you step back a bit more, and realise the VRF Name and RT/RD have pretty much no association (and then it suddenly clicks what they mean when they say "Locally Significant"...), and we - as humans - use the same VRF Name everywhere because it's easier for us, like a sort of "Poor Man's DNS for VRF RTs". So there's no reason this config wouldn't just stitch VRF "Bob" to VRF "Jane" between two Routers in the same MPLS Domain - but it'd be a pain in the arse to troubleshoot when it scaled to more than a few Routers:

Router_PE1#vrf definition Bob
 rd 192.168.0.1:999
 route-target export 65432:999
 route-target import 65432:999

Router_PE2#vrf definition Jane
 rd 192.168.0.2:999
 route-target export 65432:999
 route-target import 65432:999

Great Scott! He's got it!

Which is when it clicks - when you look at two Router's configurations and realise the RT is the same, but the RD changed; within what we've established is the same VRF "Container" (even though we renamed it across Routers, to cause pain to that guy in Ops that looked at our wife wrong during that Christmas Do, yeah - "Bob"...). So roughly then:

  • An RD can be thought of as the "Router Descriptor"
    • i.e. "Who injected that Prefix into my VRF?"
    • Probably makes sense to use a Loopback, or unique attribute of a Router; then you can jump on your Route Reflector (RR) and have a quick "Whodunnit?"
      • Router_RR1#sh ip bgp vpnv4 all | sec <Router Loopback0>:<VRF RT No>
  • An RT can be thought of as the "Routing Table Target"
    • i.e. "So that's just a VLAN-equivalent Tag for a VRF Container on the MPLS Domain then..."
    • If it's the same RT you're import/exporting everywhere, we're rocking Full Mesh; if it's not (or I'm suddenly doing loads of import statements/one export statement, or vice versa), we're looking at a pesky Hub-and-Spoke
      • Got multiple RT Import statements and one Export? You're probably on a Hub Router (for that VRF)
      • Got one RT Import statement and multiple Exports? You're probably on a Spoke Router (for that VRF)

Am I right here?

That's how I understand all this MPLS VRFery anyway; if I'm wrong, why not:

  • Tweet me @notworkd and tell me "U iz well wrong, Bruv..."
  • Write a comment below and tell me "Dude, do you even MPLS, Bro?"

 

I'm not Technical, but...

Sunday, 22 Jul 2018

A day in the life of

There you are, describing the latest solution/thesis you have to a problem, Project or task, and then someone comes along and says:

I'm not technical, but...

And just like that, you're thrown sideways. Disparaged. Condescended. Belittled. Siderailed.

My problem here isn't the content that's about to follow - it could well be valid (it could well not, too), and could change the direction of the idea to the right direction. My problem is the derision and disdainful manner this is normally delivered to me in. I mean, why even include the prelude and the "but"; if you've got something to contribute, jump in - it's what you'd do in a normal business conversation

You wouldn't say that to a Doctor

Would you say the same to another field or vocation that you were more familiar with - maybe a Doctor or Healthcare Professional? What about another field you popularly know to exist, but probably don't have exposure to - maybe a Nuclear Physicist?

It's not that your opinion is invalid; you have the right to opine about anything, in the same way I would. It's that your presentation of said opinion is disrespectful - not just of me, but also of:

  • My chosen career profession
  • The financial and personal background
  • The context of your and my employment
  • Our respective positions to our employer

But for some reason, you think that the inclusion of this prelude with a little "but" counteracts all these.

Respect that I'm here for a reason

  • I'm not a Project Manager; I'm not paid to manage time and cost of delivery (but I can cost-up and cook a three-course meal)
  • I'm not a Business Analyst; I'm not paid to analyse business requirements against our employer's strategy (but I can work out the difference to my family of buying a shiny new MacBook vs getting that larger house we could all do with)
  • I'm not a Commercial Analyst; I'm not paid to understand common costings or understanding market economics (but I do recognise that buying artisanal gluten-free bread should cost more than a standard white loaf)
  • I'm not a Legal Consultant; I'm not paid to understand the laws my employer is subject to, or loopholes within them (but I do recognise that torrenting Adobe Photoshop is a lesser crime than killing a man)

We all know how to do things we're not paid to do; that's why opinions can be valid. We're not paid to know all the things we can do; that's why you need to respect that I exist here for a reason, the same way I respect you exist.

If you don't understand or accept something, feel free to question and interject; but don't add an insulting prelude before you do, and recognise that the person you're talking to was not only employed to do the very thing you're about to question, but has also legitimately built a career upon it.

I'm not business; but...

Butt out.

Use fallocate and SCP to quickly test Cisco network throughput

Monday, 16 Apr 2018

A rubber band, a paper clip and a drinking straw...

In a pinch where you can't use iPerf to test network throughput - either because you can't access/RDP onto a Windows/Linux host, or maybe can't download iPerf from the big bad Interwebs?  I've been here before, and armed with the following tools (like a Network MacGyver), you can use Secure Copy (the FTP of the SSH world) and a bit of Linux standard binary to do much the same job:

  • Cisco (or Juniper) Switch or Router
    • RW/Admin/Priv 15 access to write a bit of config
  • Linux Box (for fallocate binary)
  • Network connectivity between said Linux Box and Cisco Router/Switch

Method

On the Linux Box

  1. Login to your Linux Box
  2. Issue the following command to create a 1 GB test file called "1gb-test.bin":
    1. fallocate -l 1G 1gb-test.bin
  3. Check your file is 1 GB in size:
    1. ls -l 1gb-test.bin

On the Cisco Switch/Router

  1. Login to your Cisco Router or Switch
  2. Enable SCP File Copy:
    1. # Cisco ASA Firewalls
      conf t
      scp copy enable
      end
      
      # Cisco Switches/Routers
      conf t
      ip scp server enable
      end
  3. (If ASA Firewall) Check your Management Firewall Rules allow SCP (TCP/22) transfer to the Cisco Switch/Router
  4. Get the Disk identifier for your SD Card/NVRAM (usually it's "disk0:" or "bootflash:") and check you have enough free space (1 GB in this example, or 1,073,741,824 bytes) for the file transfer:
    1. dir

Back on the Linux Box

  1. Copy your 1 GB test file "1gb-test.bin" from the Linux Box onto the Cisco Switch/Router/Firewall (in this example, my Router is running Disk0: as the NVRAM/Flash Volume):
    1. scp -v 1gb-test.bin <USERNAME>@<CISCO_IP_ADDRESS>:disk0:1gb-test.bin
  2. Watch the SCP File Transfer statistics, and/or...

Back on the Cisco Switch/Router

  1. Check the tx/rx rate on the interface you are expecting traffic to come in on:
    1. sh int | i Interface|Desc|rate|tx|rx
  2. (ProTip) If you've not change the stock setting, it's only sampled every 5 minutes; change that to 30 seconds (or similarly more-frequent) with:
    1. conf t
      interface <X>/<Y>
       load-interval 30
      end

Results

That's it; if you're using the correct interface you are bothered about, you are now doing a TCP/22 (SSH/SCP) transfer of a 1 GB test file from a Linux Box to your Cisco Router/Firewall/Switch. Bear in mind that it might not be the throughput rate you are expecting (or what the LAN/WAN Link can actually perform at), due to a few limiting factors:

  • NVRAM/Flash Medium transfer speed (microSD Cards in ASR's are faster than, say, CompactFlash in older ISR's)
  • CoPP (Control Plane Policing)
    • Technically, SCP is Control Plane operation to the Router/Switch rather than Data Plane through the Router/Switch, so your SCP copy might be being rate-limited by this

Don't forget to delete your test file when you're done, and note if the file copy doesn't complete, the file isn't pre-allocated on IOS - so a subsequent "dir" will show no data written to disk.

I've personally found this useful for minimal "stress-testing", or to try and invoke some legitimate LAN/WAN traffic to show up on a NetFlow Collector or SNMP Polling NMS (maybe something like LibreNMS).

Just tweak the setting on the doofer...

Saturday, 14 Apr 2018

Welcome

Welcome to notworkd.io - the newest (probably irrelevant) Networking, Cloud, SDN, Technology blog by a bloke employed to make all this stuff work, but who normally finds himself as more of a Notwork Engineer than a Network Engineer.

My hope in making this blog is to help out anyone in the same boat as me, who finds themselves working for a larger company (where various Technologies and Vendors are thrust upon us, rather than bring coded in-house or self-chosen) - and to act as a a bit of a FAQ/How To, for the various stuff I'll be talking about. Ditto, I'd like to get a bit more involved in "the Community", rather than just being a ardent lurker; so doing this gives me a nice platform to contribute something back (quite what the worth of that will be, I'm not yet sure).

Things I reckon I'll talk about

A short list of things I hope I'll be talking about in coming posts and months:

  • Enterprise Networking
  • Service Provider Networking
  • Struggling to get to grips with Cloud stuff (I thought vPC was a Cisco thing, not an AWS thing...)
  • Juniper Firewalls and Routers
  • Cisco Routers, Switches, Firewalls, Wireless LAN Controllers, ISE Nodes, NAC, other gubbins
  • A dislike for the speed I can't do Visio diagrams at
  • Learning about SysAdmin and SysOps Linux black magic (SSH Tunneling - We proxied your TCP through an SSH Tunnel, so you can Proxy while you Proxy your Proxy, Bro)
  • Documenting those pesky Gotchas with pretty much every Vendor's newest "Never Breaks this, honest 'Guv" feature

Fancy getting in touch

Give me a Tweet @notworkd, or drop a comment on here.

Let's get the party started

Hmm, as soon as I find the "Play" button on this bad boy, I reckon we're in business with this Cloud jazz. Must be around here somewhere...

 undefined

Home